Jul 13 18:47:42.121379 ip-10-0-131-203 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 13 18:47:42.121391 ip-10-0-131-203 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 13 18:47:42.121400 ip-10-0-131-203 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 13 18:47:42.121626 ip-10-0-131-203 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 13 18:47:52.121717 ip-10-0-131-203 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 13 18:47:52.121736 ip-10-0-131-203 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ce88f194952d4d7584a0af620cd9c3a7 --
Jul 13 18:49:22.970090 ip-10-0-131-203 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 18:49:23.412496 ip-10-0-131-203 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:49:23.412496 ip-10-0-131-203 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 18:49:23.412496 ip-10-0-131-203 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:49:23.412496 ip-10-0-131-203 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 18:49:23.412496 ip-10-0-131-203 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:49:23.414599 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.414509    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 18:49:23.420178 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420163    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:23.420178 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420178    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420182    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420186    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420189    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420193    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420201    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420204    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420206    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420209    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420211    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420214    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420216    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420219    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420222    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420224    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420227    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420229    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420232    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420234    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420237    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:23.420243 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420240    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420242    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420245    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420247    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420250    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420253    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420255    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420258    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420261    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420263    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420266    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420268    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420271    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420273    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420276    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420278    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420281    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420284    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420287    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:23.420710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420289    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420293    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420297    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420299    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420302    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420304    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420307    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420309    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420312    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420314    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420338    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420342    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420345    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420348    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420351    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420355    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420358    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420360    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420363    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:23.421214 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420366    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420368    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420373    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420376    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420379    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420382    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420386    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420389    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420392    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420395    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420398    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420400    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420403    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420406    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420408    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420411    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420413    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420415    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420418    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:23.421668 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420420    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420423    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420425    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420428    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420430    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420432    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420435    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420438    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420440    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420443    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420447    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420449    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420452    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420454    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420457    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420459    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420462    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420464    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420466    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420469    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:23.422143 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.420471    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421213    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421220    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421223    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421226    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421229    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421233    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421236    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421239    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421241    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421244    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421247    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421249    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421252    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421254    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421256    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421259    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421261    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421264    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421266    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:23.422660 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421269    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421272    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421275    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421277    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421280    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421283    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421285    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421288    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421290    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421292    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421295    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421297    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421300    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421302    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421305    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421307    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421309    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421312    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421315    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421318    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:23.423173 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421320    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421323    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421325    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421327    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421330    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421332    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421335    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421339    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421342    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421345    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421348    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421351    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421354    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421356    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421359    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421361    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421364    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421366    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421369    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:23.423753 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421371    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421374    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421376    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421378    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421381    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421383    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421386    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421388    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421390    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421392    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421396    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421399    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421402    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421405    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421408    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421410    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421413    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421417    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421419    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421422    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:23.424282 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421425    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421427    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421430    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421432    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421435    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421439    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421442    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421444    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421447    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421449    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421452    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421454    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421456    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421459    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421461    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421464    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421468    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421470    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421473    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:23.424888 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421476    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421478    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421892    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421898    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421901    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421904    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421907    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421911    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421915    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421918    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421920    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421923    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421925    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421928    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421931    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421933    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421935    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421938    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421941    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:23.425899 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421944    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421946    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421949    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421951    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421954    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421957    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421959    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421961    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421964    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421966    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421969    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421971    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421973    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421976    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421978    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421980    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421983    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421985    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421988    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421990    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:23.426445 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421993    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421995    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.421998    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422000    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422002    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422005    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422010    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422013    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422016    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422019    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422022    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422026    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422029    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422031    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422034    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422036    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422039    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422041    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422044    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422046    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:23.426939 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422048    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422051    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422054    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422056    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422059    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422061    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422064    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422066    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422068    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422071    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422073    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422075    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422078    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422081    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422083    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422086    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422088    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422091    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422095    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422097    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:23.427432 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422100    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422102    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422105    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422107    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422110    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422115    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422117    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422121    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422124    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422127    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422130    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422132    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422135    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422137    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422140    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422142    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422144    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422147    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422150    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:23.427947 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422152    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422154    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422157    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422223    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422230    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422236    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422240    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422244    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422247    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422251    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422255    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422260    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422263    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422267    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422270    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422273    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422278    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422280    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422283    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422286    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422289    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422293    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422296    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422299    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 18:49:23.428402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422302    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422305    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422309    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422311    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422315    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422318    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422321    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422323    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422326    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422329    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422332    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422336    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422339    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422341    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422344    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422347    2581 flags.go:64] FLAG: --enable-server="true"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422350    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422354    2581 flags.go:64] FLAG: --event-burst="100"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422357    2581 flags.go:64] FLAG: --event-qps="50"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422361    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422364    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422367    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422371    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422373    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422378    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 18:49:23.428988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422380    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422383    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422386    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422389    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422392    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422395    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422398    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422401    2581 flags.go:64] FLAG: --feature-gates=""
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422405    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422408    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422411    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422413    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422416    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422419    2581 flags.go:64] FLAG: --help="false"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422422    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422425    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422427    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422430    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422433    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422436    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422439    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422442    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422444    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 18:49:23.429577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422447    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422450    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422453    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422457    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422460    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422463    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422466    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422469    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422472    2581 flags.go:64] FLAG: --lock-file=""
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422475    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422478    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422481    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422486    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422489    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422491    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422494    2581 flags.go:64] FLAG: --logging-format="text"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422497    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422500    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422503    2581 flags.go:64] FLAG: --manifest-url=""
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422505    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422510    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422512    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422516    2581 flags.go:64] FLAG: --max-pods="110"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422519    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422521    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 18:49:23.430170 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422524    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422527    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422530    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422541    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422544    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422550    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422553    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422556    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422559    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422562    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422568    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422572    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422575    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422578    2581 flags.go:64] FLAG: --port="10250"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422581    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422585    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e8f7711c3448be5e"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422588    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422591    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422593    2581 flags.go:64] FLAG: --register-node="true"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422596    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422600    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422602    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422605    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422608    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422611    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 18:49:23.430775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422614    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422617    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422619    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422622    2581 flags.go:64] FLAG: --runonce="false"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422625    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422628    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422631    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422633    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422636    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422639    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422642    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422645    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422648    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422650    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422653    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422655    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422658    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422662    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422666    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422671    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422674    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422677    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422681    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422684    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422687    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 18:49:23.431440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422690    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422692    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422695    2581 flags.go:64] FLAG: --v="2"
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422699    2581 flags.go:64] FLAG: --version="false"
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422703    2581 flags.go:64] FLAG: --vmodule=""
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422707    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.422710    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422838    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422842    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422845    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422848    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422851    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422853    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422856    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422858    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422861    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422863    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422867    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422869    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422872    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422874    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:23.432069 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422877    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422879    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422881    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422884    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422889    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422892    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422894    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422897    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422900    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422902    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422905    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422907    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422910    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422912    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422915    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422917    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422920    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422922    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422924    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:23.432566 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422927    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422929    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422932    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422934    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422937    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422940    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422942    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422944    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422947    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422949    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422952    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422954    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422957    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422960    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422962    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422964    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422967    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422970    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422973    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422976    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:23.433042 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422978    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422981    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422983    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422985    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422988    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422990    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422993    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422995    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.422998    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423000    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423003    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423005    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423007    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423010    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423012    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423015    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423018    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423020    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423023    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423025    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:23.433527 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423028    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423030    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423032    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423036    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423039    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423042    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423045    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423048    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423050    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423055    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423058    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423063    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423067    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423070    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423073    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423075    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423078    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423081    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423083    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:23.434062 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423086    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:23.434524 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423088    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:23.434524 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423090    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:23.434524 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423093    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:23.434524 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423095    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:23.434524 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423098    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:23.434524 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.423100    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:23.434524 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.423882    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:49:23.439934 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.439915    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 18:49:23.439934 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.439934    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 18:49:23.440012 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.439952    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 18:49:23.440012 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.439958    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440015    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440020    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440024    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440027    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440030    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440033    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440035    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440038    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440041    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440044    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440046    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440049    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440052    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440055    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440057    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440060    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440062    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440065    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440067    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:23.440065 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440070    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440073    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440075    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440078    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440082    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440086    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440089    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440093    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440096    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440099    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440102    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440105    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440108    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440110    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440113    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440116    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440118    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440120    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440123    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:23.440541 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440126    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440128    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440131    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440134    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440136    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440139    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440141    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440144    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440147    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440150    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440152    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440155    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440157    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440159    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440162    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440164    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440167    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440170    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440172    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440175    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:23.441008 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440178    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440181    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440183    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440186    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440188    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440191    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440193    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440195    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440198    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440200    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440203    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440205    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440207    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440210    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440213    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440215    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440218    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440220    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440223    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440225    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:23.441494 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440228    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440231    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440233    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440236    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440238    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440240    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440242    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440245    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440248    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440251    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440253    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440255    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440258    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440260    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440262    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440265    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440267    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440270    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440272    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440274    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:23.441981 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440276    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.440281    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440395    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440400    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440403    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440405    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440409    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440411    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440415    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440418    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440421    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440424    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440426    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440430    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:23.442503 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440432    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440435    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440437    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440440    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440442    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440445    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440447    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440450    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440452    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440455    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440457    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440460    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440462    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440464    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440467    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440469    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440471    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440474    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440476    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440478    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:23.443002 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440481    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440483    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440485    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440487    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440490    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440492    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440494    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440497    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440499    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440501    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440504    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440506    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440509    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440511    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440513    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440516    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440518    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440521    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440523    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440525    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:23.443472 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440527    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440530    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440532    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440534    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440537    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440539    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440542    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440544    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440547    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440550    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440552    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440555    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440557    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440559    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440562    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440564    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440567    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440569    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440572    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440574    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:23.443965 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440576    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440578    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440581    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440583    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440587    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440590    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440593    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440596    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440598    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440601    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440604    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440606    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440609    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440611    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440614    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440616    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440618    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440621    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440623    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:23.444509 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440625    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440628    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440630    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440632    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440635    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440637    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440639    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:23.440641    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.440647    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.440742    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 18:49:23.444986 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.444125    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 18:49:23.445958 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.445947    2581 server.go:1019] "Starting client certificate rotation"
Jul 13 18:49:23.446083 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.446064    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:49:23.446142 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.446116    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:49:23.450978 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.450904    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:49:23.452212 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.452196    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:49:23.463586 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.463571    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 13 18:49:23.466169 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.466154    2581 log.go:25] "Validated CRI v1 image API"
Jul 13 18:49:23.466943 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.466929    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 18:49:23.470322 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.470300    2581 fs.go:135] Filesystem UUIDs: map[73ff6cb5-c67c-4338-99a1-a40a38a314ab:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b2f84723-ddbe-4674-9bb0-c1f0a6b3f1fa:/dev/nvme0n1p3]
Jul 13 18:49:23.470382 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.470321    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 18:49:23.474751 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.474641    2581 manager.go:216] Machine: {Timestamp:2026-07-13 18:49:23.474147526 +0000 UTC m=+0.374372909 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099787 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec251e7a95c8645a4e5e657c54497d9c SystemUUID:ec251e7a-95c8-645a-4e5e-657c54497d9c BootID:ce88f194-952d-4d75-84a0-af620cd9c3a7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:22:15:35:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:22:15:35:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:98:0c:39:9e:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 18:49:23.474751 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.474741    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 18:49:23.474895 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.474869    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 18:49:23.475291 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.475273    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:49:23.476078 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.476050    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 18:49:23.476216 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.476080    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-203.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 18:49:23.477663 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.477652    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 18:49:23.477703 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.477665    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 18:49:23.477703 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.477673    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:49:23.478477 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.478467    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:49:23.480517 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.480506    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 18:49:23.485071 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485059    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:49:23.485206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485196    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 18:49:23.485250 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485228    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 18:49:23.485250 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485245    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 18:49:23.485313 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485258    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 18:49:23.485313 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485269    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 13 18:49:23.485313 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485284    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 18:49:23.485781 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485760    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:49:23.485910 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.485804    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:49:23.489590 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.489558    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 18:49:23.492111 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492093    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 18:49:23.492201 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492132    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 18:49:23.492359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492346    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 18:49:23.492427 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492366    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 18:49:23.492427 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492375    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 18:49:23.492427 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492383    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 18:49:23.492427 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492395    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 18:49:23.492427 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492402    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 18:49:23.492989 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492977    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 18:49:23.493040 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.492991    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 18:49:23.493040 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.493001    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 18:49:23.493040 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.493010    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 18:49:23.493040 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.493022    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 18:49:23.493040 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.493034    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 18:49:23.493241 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.493066    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 18:49:23.493241 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.493076    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 18:49:23.496374 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.496357    2581 server.go:1268] "Started kubelet"
Jul 13 18:49:23.496584 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.496531    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 18:49:23.496682 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.496475    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 18:49:23.496740 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.496716    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 18:49:23.497285 ip-10-0-131-203 systemd[1]: Started Kubernetes Kubelet.
Jul 13 18:49:23.498301 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.498099    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 18:49:23.498354 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.498111    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 18:49:23.498354 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.498302    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 18:49:23.498354 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.498197    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-203.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 18:49:23.498831 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.498816    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 18:49:23.502599 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.502579    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 18:49:23.505023 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.503711    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-203.ec2.internal.18c1edf4e5a9a366  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-203.ec2.internal,UID:ip-10-0-131-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-203.ec2.internal,},FirstTimestamp:2026-07-13 18:49:23.496330086 +0000 UTC m=+0.396555468,LastTimestamp:2026-07-13 18:49:23.496330086 +0000 UTC m=+0.396555468,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-203.ec2.internal,}"
Jul 13 18:49:23.506208 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.506190    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 18:49:23.506267 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.506208    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 18:49:23.506306 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.506296    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:23.507022 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.506969    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 18:49:23.507022 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.506971    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 18:49:23.507022 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507015    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 18:49:23.507222 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507092    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 18:49:23.507222 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507105    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 18:49:23.507222 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507125    2581 factory.go:55] Registering systemd factory
Jul 13 18:49:23.507222 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507177    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 13 18:49:23.507525 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507511    2581 factory.go:153] Registering CRI-O factory
Jul 13 18:49:23.507592 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507528    2581 factory.go:223] Registration of the crio container factory successfully
Jul 13 18:49:23.507625 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507616    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 18:49:23.507655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507638    2581 factory.go:103] Registering Raw factory
Jul 13 18:49:23.507655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.507652    2581 manager.go:1196] Started watching for new ooms in manager
Jul 13 18:49:23.508166 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.508150    2581 manager.go:349] Starting recovery of all containers
Jul 13 18:49:23.511629 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.511606    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 18:49:23.511712 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.511670    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 18:49:23.518965 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.518942    2581 manager.go:354] Recovery completed
Jul 13 18:49:23.521293 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.521226    2581 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 13 18:49:23.524090 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.524077    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:23.526701 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.526681    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f7r57"
Jul 13 18:49:23.526818 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.526747    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:23.526818 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.526780    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:23.526818 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.526812    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:23.527359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.527344    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 18:49:23.527359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.527350    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:23.527359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.527357    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 18:49:23.527513 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.527375    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:49:23.528211 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.528189    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:23.528299 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.528219    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:23.528299 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.528232    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:23.528959 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.528876    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-203.ec2.internal.18c1edf4e779fbd8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-203.ec2.internal,UID:ip-10-0-131-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-203.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-203.ec2.internal,},FirstTimestamp:2026-07-13 18:49:23.526761432 +0000 UTC m=+0.426986819,LastTimestamp:2026-07-13 18:49:23.526761432 +0000 UTC m=+0.426986819,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-203.ec2.internal,}"
Jul 13 18:49:23.530106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.530091    2581 policy_none.go:49] "None policy: Start"
Jul 13 18:49:23.530159 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.530109    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 18:49:23.530159 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.530119    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 18:49:23.532164 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.532151    2581 policy_none.go:47] "Start"
Jul 13 18:49:23.532265 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.532248    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f7r57"
Jul 13 18:49:23.536411 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.536340    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-203.ec2.internal.18c1edf4e77a9ef3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-203.ec2.internal,UID:ip-10-0-131-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-203.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-203.ec2.internal,},FirstTimestamp:2026-07-13 18:49:23.526803187 +0000 UTC m=+0.427028572,LastTimestamp:2026-07-13 18:49:23.526803187 +0000 UTC m=+0.427028572,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-203.ec2.internal,}"
Jul 13 18:49:23.584534 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.584518    2581 manager.go:337] "Starting Device Plugin manager"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.584603    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.584628    2581 server.go:87] "Starting device plugin registration server"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.584910    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.584923    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.585163    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.585252    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.585260    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.585663    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.585711    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.595780    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.597134    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.597167    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.597187    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.597218    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 18:49:23.602278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.599490    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:23.685927 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.685850    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:23.686965 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.686946    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:23.687070 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.686976    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:23.687070 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.686989    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:23.687070 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.687014    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.696436 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.696413    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.696529 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.696440    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-203.ec2.internal\": node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:23.697634 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.697608    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal"]
Jul 13 18:49:23.697684 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.697678    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:23.700115 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.700098    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:23.700206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.700131    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:23.700206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.700145    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:23.702490 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.702477    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:23.702589 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.702575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.702633 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.702604    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:23.703189 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.703165    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:23.703189 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.703181    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:23.703284 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.703191    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:23.703284 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.703201    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:23.703284 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.703206    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:23.703284 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.703210    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:23.705433 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.705416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.705543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.705441    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:23.706093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.706075    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:23.706181 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.706104    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:23.706181 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.706114    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:23.708080 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.708064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65659a82159918e78dc6edc085316252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal\" (UID: \"65659a82159918e78dc6edc085316252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.708151 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.708087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65659a82159918e78dc6edc085316252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal\" (UID: \"65659a82159918e78dc6edc085316252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.708151 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.708104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58f7c83942b70dd267414f6b25e3812d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-203.ec2.internal\" (UID: \"58f7c83942b70dd267414f6b25e3812d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.721067 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.721041    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:23.724473 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.724454    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-203.ec2.internal\" not found" node="ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.728825 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.728810    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-203.ec2.internal\" not found" node="ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.809016 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.808989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65659a82159918e78dc6edc085316252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal\" (UID: \"65659a82159918e78dc6edc085316252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.809016 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.809018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65659a82159918e78dc6edc085316252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal\" (UID: \"65659a82159918e78dc6edc085316252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.809168 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.809035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58f7c83942b70dd267414f6b25e3812d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-203.ec2.internal\" (UID: \"58f7c83942b70dd267414f6b25e3812d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.809168 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.809061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58f7c83942b70dd267414f6b25e3812d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-203.ec2.internal\" (UID: \"58f7c83942b70dd267414f6b25e3812d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.809168 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.809073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65659a82159918e78dc6edc085316252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal\" (UID: \"65659a82159918e78dc6edc085316252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.809168 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:23.809090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65659a82159918e78dc6edc085316252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal\" (UID: \"65659a82159918e78dc6edc085316252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:23.822149 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.822121    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:23.923198 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:23.923159    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.024081 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.024003    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.027171 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.027155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:24.032585 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.032395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:24.125070 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.125038    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.225546 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.225511    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.326149 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.326080    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.426659 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.426631    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.440095 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.440072    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:24.444482 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.444459    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 18:49:24.444597 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.444576    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:49:24.444656 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.444577    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:49:24.527072 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.527041    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.537213 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.537178    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 18:44:24 +0000 UTC" deadline="2028-04-23 11:12:35.39514995 +0000 UTC"
Jul 13 18:49:24.537213 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.537211    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15592h23m10.857941963s"
Jul 13 18:49:24.541686 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:24.541653    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f7c83942b70dd267414f6b25e3812d.slice/crio-9c0f3348b6312beacd5a42bad18962b13a8a7cb0c68d229552a30d42db3951e6 WatchSource:0}: Error finding container 9c0f3348b6312beacd5a42bad18962b13a8a7cb0c68d229552a30d42db3951e6: Status 404 returned error can't find the container with id 9c0f3348b6312beacd5a42bad18962b13a8a7cb0c68d229552a30d42db3951e6
Jul 13 18:49:24.542184 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:24.542143    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65659a82159918e78dc6edc085316252.slice/crio-04cab46ad216a8153aba32a7959588282add6e6b5794f11b8565c7a1b176eb99 WatchSource:0}: Error finding container 04cab46ad216a8153aba32a7959588282add6e6b5794f11b8565c7a1b176eb99: Status 404 returned error can't find the container with id 04cab46ad216a8153aba32a7959588282add6e6b5794f11b8565c7a1b176eb99
Jul 13 18:49:24.547511 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.547491    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:49:24.550153 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.550137    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:24.550853 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.550839    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:24.550915 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.550869    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:24.550915 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.550882    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:24.600887 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.600804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal" event={"ID":"58f7c83942b70dd267414f6b25e3812d","Type":"ContainerStarted","Data":"9c0f3348b6312beacd5a42bad18962b13a8a7cb0c68d229552a30d42db3951e6"}
Jul 13 18:49:24.601781 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.601761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal" event={"ID":"65659a82159918e78dc6edc085316252","Type":"ContainerStarted","Data":"04cab46ad216a8153aba32a7959588282add6e6b5794f11b8565c7a1b176eb99"}
Jul 13 18:49:24.627270 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.627252    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.714896 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:24.714871    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:24.728316 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.728293    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.828821 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.828775    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:24.929413 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:24.929342    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:25.030439 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.030409    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-203.ec2.internal\" not found"
Jul 13 18:49:25.094069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.094035    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:25.106955 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.106922    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:25.121500 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.121474    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:49:25.122932 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.122913    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal"
Jul 13 18:49:25.131644 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.131618    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:49:25.486730 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.486692    2581 apiserver.go:52] "Watching apiserver"
Jul 13 18:49:25.494868 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.494630    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 18:49:25.495154 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.495127    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal","openshift-image-registry/node-ca-ql9cx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal","openshift-multus/multus-additional-cni-plugins-2rdtb","openshift-multus/multus-bgqcq","openshift-network-diagnostics/network-check-target-5l5l6","openshift-ovn-kubernetes/ovnkube-node-c2s8k","kube-system/global-pull-secret-syncer-q4wtd","kube-system/konnectivity-agent-4qmlg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp","openshift-cluster-node-tuning-operator/tuned-mbnkx","openshift-dns/node-resolver-pd2tc","openshift-multus/network-metrics-daemon-45fpx","openshift-network-operator/iptables-alerter-54jpc"]
Jul 13 18:49:25.498832 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.498805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.501539 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.501316    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 18:49:25.501539 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.501355    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 18:49:25.501724 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.501642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p96qv\""
Jul 13 18:49:25.502287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.501920    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 18:49:25.502287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.501947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 18:49:25.502287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.501948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 18:49:25.502287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.502050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 18:49:25.502689 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.502669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:25.507387 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.502761    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:25.507387 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.506044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.507387 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.506625    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 18:49:25.508229 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.508200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p699b\""
Jul 13 18:49:25.508532 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.508511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 18:49:25.508854 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.508827    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 18:49:25.509105 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.509086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:49:25.510948 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.510924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.513381 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.513360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.514551 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.514530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:25.514649 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.514612    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:25.516654 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.516633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 18:49:25.516903 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.516886    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 18:49:25.517205 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.517187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9gbfj\""
Jul 13 18:49:25.517511 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.517494    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 18:49:25.517746 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.517707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qgxdh\""
Jul 13 18:49:25.517982 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.517967    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 18:49:25.518296 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.518281    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 18:49:25.518516 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.518503    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 18:49:25.518750 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.518736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.521431 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.521410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.521531 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.521492    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:25.522045 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-var-lib-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522161 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-node-log\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522161 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522161 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.522329 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovnkube-script-lib\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522329 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-kubelet\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522329 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovnkube-config\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522329 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qm96\" (UniqueName: \"kubernetes.io/projected/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-kube-api-access-2qm96\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:25.522329 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-system-cni-dir\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.522329 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-run-netns\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-ovn\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-log-socket\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-cni-netd\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntvp7\" (UniqueName: \"kubernetes.io/projected/b0f5f16f-b1dc-44be-b416-da33fca52c24-kube-api-access-ntvp7\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-systemd-units\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wd77\" (UniqueName: \"kubernetes.io/projected/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-kube-api-access-9wd77\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-cni-bin\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-env-overrides\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0f5f16f-b1dc-44be-b416-da33fca52c24-iptables-alerter-script\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cnibin\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.522802 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2rs9\" (UniqueName: \"kubernetes.io/projected/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-kube-api-access-d2rs9\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.523183 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-systemd\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.523183 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.523183 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovn-node-metrics-cert\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.523183 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f5f16f-b1dc-44be-b416-da33fca52c24-host-slash\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.523183 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-slash\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.523183 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.522973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-etc-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.523183 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.523008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-os-release\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.523859 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.523798    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rwd9x\""
Jul 13 18:49:25.524178 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.524161    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 18:49:25.524485 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.524454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 18:49:25.524696 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.524680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.525228 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.525208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.528493 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.528472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.529248 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.529220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 18:49:25.530191 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.530169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 18:49:25.530292 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.530196    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7v5xt\""
Jul 13 18:49:25.531685 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.531661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9prxv\""
Jul 13 18:49:25.532274 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.531773    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 18:49:25.532274 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.532013    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:49:25.532274 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.532019    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 18:49:25.532274 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.531665    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 18:49:25.532274 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.532251    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8ght8\""
Jul 13 18:49:25.532605 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.532397    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 18:49:25.532605 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.532472    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 18:49:25.533918 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.533899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.537001 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.536949    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 18:49:25.537110 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.537090    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 18:49:25.539101 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.539070    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47xtd\""
Jul 13 18:49:25.539343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.539294    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:49:25.563724 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.563695    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-psdn9"
Jul 13 18:49:25.579971 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.579949    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-psdn9"
Jul 13 18:49:25.608740 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.608715    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 18:49:25.623649 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-etc-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.623793 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-os-release\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.623793 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-socket-dir-parent\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.623793 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-k8s-cni-cncf-io\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.623793 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-node-log\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.623793 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e61056f-7549-4981-9e2f-885a7705b066-host\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-system-cni-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-cnibin\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-device-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.623973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-kubernetes\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-registration-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.624046 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovnkube-config\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-system-cni-dir\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-run\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-host\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-cni-bin\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-kubelet-config\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-sys-fs\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-tmp-dir\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-run-netns\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntvp7\" (UniqueName: \"kubernetes.io/projected/b0f5f16f-b1dc-44be-b416-da33fca52c24-kube-api-access-ntvp7\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.624371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphd6\" (UniqueName: \"kubernetes.io/projected/9e61056f-7549-4981-9e2f-885a7705b066-kube-api-access-bphd6\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-cni-multus\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-sys\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-tuned\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn7f\" (UniqueName: \"kubernetes.io/projected/e22e24ba-97a9-42e4-9bf1-16d3aed89805-kube-api-access-lcn7f\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-systemd-units\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e61056f-7549-4981-9e2f-885a7705b066-serviceca\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-daemon-config\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9m8\" (UniqueName: \"kubernetes.io/projected/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-kube-api-access-vf9m8\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-cni-bin\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-env-overrides\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0f5f16f-b1dc-44be-b416-da33fca52c24-iptables-alerter-script\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.624783 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2rs9\" (UniqueName: \"kubernetes.io/projected/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-kube-api-access-d2rs9\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-modprobe-d\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovn-node-metrics-cert\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysctl-conf\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-os-release\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.624982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-kubelet\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6e212d7-c36c-4e94-86b9-c34e88752fce-konnectivity-ca\") pod \"konnectivity-agent-4qmlg\" (UID: \"f6e212d7-c36c-4e94-86b9-c34e88752fce\") " pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-dbus\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-slash\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkv6v\" (UniqueName: \"kubernetes.io/projected/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-kube-api-access-vkv6v\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-hosts-file\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-var-lib-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-socket-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.625333 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q686c\" (UniqueName: \"kubernetes.io/projected/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-kube-api-access-q686c\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovnkube-script-lib\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-cni-binary-copy\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-kubelet\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qm96\" (UniqueName: \"kubernetes.io/projected/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-kube-api-access-2qm96\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysctl-d\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-var-lib-kubelet\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-cni-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-ovn\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-log-socket\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-cni-netd\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-hostroot\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22e24ba-97a9-42e4-9bf1-16d3aed89805-tmp\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wd77\" (UniqueName: \"kubernetes.io/projected/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-kube-api-access-9wd77\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:25.626065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysconfig\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-systemd\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-netns\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-multus-certs\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.625835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cnibin\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.626524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.626615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-etc-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.626690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-os-release\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.626744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-node-log\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.626824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.626791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.627254 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.626958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.627579 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.627551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovnkube-config\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.627699 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.627630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-system-cni-dir\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.627768 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.627745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-run-netns\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.627863 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.627842    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:25.627980 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.627932    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:49:26.127909293 +0000 UTC m=+3.028134669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:25.628414 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.628393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-systemd-units\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.628492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.628543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-cni-bin\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.629368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0f5f16f-b1dc-44be-b416-da33fca52c24-iptables-alerter-script\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.629828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.630148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.630475    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.630835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-kubelet\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-slash\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-host-cni-netd\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-ovn\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-var-lib-openvswitch\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-log-socket\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-cnibin\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-conf-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-etc-kubernetes\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6e212d7-c36c-4e94-86b9-c34e88752fce-agent-certs\") pod \"konnectivity-agent-4qmlg\" (UID: \"f6e212d7-c36c-4e94-86b9-c34e88752fce\") " pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.631998 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-systemd\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.632911 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f5f16f-b1dc-44be-b416-da33fca52c24-host-slash\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.632911 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-lib-modules\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.632911 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f5f16f-b1dc-44be-b416-da33fca52c24-host-slash\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.632911 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.631951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-run-systemd\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.633124 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.633073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-env-overrides\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.633329 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.633287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovnkube-script-lib\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.634914 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.634893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-ovn-node-metrics-cert\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.648820 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.648794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntvp7\" (UniqueName: \"kubernetes.io/projected/b0f5f16f-b1dc-44be-b416-da33fca52c24-kube-api-access-ntvp7\") pod \"iptables-alerter-54jpc\" (UID: \"b0f5f16f-b1dc-44be-b416-da33fca52c24\") " pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.653771 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.653747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2rs9\" (UniqueName: \"kubernetes.io/projected/96e98cef-d1a2-4caf-af2b-22ea55efc5ac-kube-api-access-d2rs9\") pod \"multus-additional-cni-plugins-2rdtb\" (UID: \"96e98cef-d1a2-4caf-af2b-22ea55efc5ac\") " pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.665125 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.665103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qm96\" (UniqueName: \"kubernetes.io/projected/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-kube-api-access-2qm96\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:25.665424 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.665402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wd77\" (UniqueName: \"kubernetes.io/projected/cf537d94-8daa-47b7-9c7e-02e3c5e25f0c-kube-api-access-9wd77\") pod \"ovnkube-node-c2s8k\" (UID: \"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.732432 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysctl-conf\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.732610 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-os-release\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.732610 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-kubelet\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.732610 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-os-release\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.732610 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6e212d7-c36c-4e94-86b9-c34e88752fce-konnectivity-ca\") pod \"konnectivity-agent-4qmlg\" (UID: \"f6e212d7-c36c-4e94-86b9-c34e88752fce\") " pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.732824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-kubelet\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.732824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysctl-conf\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.732824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-dbus\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.732824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.732824 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkv6v\" (UniqueName: \"kubernetes.io/projected/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-kube-api-access-vkv6v\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-hosts-file\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-socket-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q686c\" (UniqueName: \"kubernetes.io/projected/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-kube-api-access-q686c\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-cni-binary-copy\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.732995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysctl-d\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-var-lib-kubelet\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-cni-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-hostroot\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22e24ba-97a9-42e4-9bf1-16d3aed89805-tmp\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6e212d7-c36c-4e94-86b9-c34e88752fce-konnectivity-ca\") pod \"konnectivity-agent-4qmlg\" (UID: \"f6e212d7-c36c-4e94-86b9-c34e88752fce\") " pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysconfig\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-systemd\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-netns\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-multus-certs\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-conf-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-etc-kubernetes\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6e212d7-c36c-4e94-86b9-c34e88752fce-agent-certs\") pod \"konnectivity-agent-4qmlg\" (UID: \"f6e212d7-c36c-4e94-86b9-c34e88752fce\") " pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-lib-modules\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-socket-dir-parent\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-cni-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-k8s-cni-cncf-io\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e61056f-7549-4981-9e2f-885a7705b066-host\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-system-cni-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-cnibin\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.733552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-dbus\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-device-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.733581    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-kubernetes\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-conf-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.733625    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret podName:c020b2f3-f25b-4d53-8e98-360ac0d1ede6 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:26.233610898 +0000 UTC m=+3.133836270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret") pod "global-pull-secret-syncer-q4wtd" (UID: "c020b2f3-f25b-4d53-8e98-360ac0d1ede6") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-registration-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-hostroot\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-registration-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-hosts-file\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.733916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-socket-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e61056f-7549-4981-9e2f-885a7705b066-host\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-lib-modules\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-socket-dir-parent\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734359 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-k8s-cni-cncf-io\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-cnibin\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-device-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-var-lib-kubelet\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-kubernetes\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysconfig\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-sysctl-d\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-etc-kubernetes\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-systemd\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-multus-certs\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-system-cni-dir\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-run-netns\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-run\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-host\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-run\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-cni-bin\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.734857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-kubelet-config\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-sys-fs\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-cni-bin\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-kubelet-config\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-tmp-dir\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-host\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-sys-fs\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphd6\" (UniqueName: \"kubernetes.io/projected/9e61056f-7549-4981-9e2f-885a7705b066-kube-api-access-bphd6\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-cni-multus\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-sys\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-tuned\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn7f\" (UniqueName: \"kubernetes.io/projected/e22e24ba-97a9-42e4-9bf1-16d3aed89805-kube-api-access-lcn7f\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.734999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e61056f-7549-4981-9e2f-885a7705b066-serviceca\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.735012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-daemon-config\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.735028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9m8\" (UniqueName: \"kubernetes.io/projected/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-kube-api-access-vf9m8\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.735049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-modprobe-d\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.735129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-modprobe-d\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.735155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-host-var-lib-cni-multus\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.735543 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.735222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22e24ba-97a9-42e4-9bf1-16d3aed89805-sys\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.736600 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.736127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-tmp-dir\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.736600 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.736445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e61056f-7549-4981-9e2f-885a7705b066-serviceca\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.736600 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.736476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-multus-daemon-config\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.739314 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.739256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-cni-binary-copy\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.739425 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.739374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22e24ba-97a9-42e4-9bf1-16d3aed89805-tmp\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.739599 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.739577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6e212d7-c36c-4e94-86b9-c34e88752fce-agent-certs\") pod \"konnectivity-agent-4qmlg\" (UID: \"f6e212d7-c36c-4e94-86b9-c34e88752fce\") " pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.741425 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.741403    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:25.741546 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.741437    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:25.741546 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.741455    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xmchq for pod openshift-network-diagnostics/network-check-target-5l5l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:25.741546 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:25.741508    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq podName:9748f76c-1486-4756-90f7-4ab1d24bcc11 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:26.24149029 +0000 UTC m=+3.141715664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xmchq" (UniqueName: "kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq") pod "network-check-target-5l5l6" (UID: "9748f76c-1486-4756-90f7-4ab1d24bcc11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:25.741854 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.741804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e22e24ba-97a9-42e4-9bf1-16d3aed89805-etc-tuned\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.743932 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.743908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q686c\" (UniqueName: \"kubernetes.io/projected/2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf-kube-api-access-q686c\") pod \"node-resolver-pd2tc\" (UID: \"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf\") " pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:25.744159 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.744137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkv6v\" (UniqueName: \"kubernetes.io/projected/b63f3d9e-26c5-4199-b3f1-402a91a4e5ed-kube-api-access-vkv6v\") pod \"aws-ebs-csi-driver-node-mmtjp\" (UID: \"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.746265 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.746242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphd6\" (UniqueName: \"kubernetes.io/projected/9e61056f-7549-4981-9e2f-885a7705b066-kube-api-access-bphd6\") pod \"node-ca-ql9cx\" (UID: \"9e61056f-7549-4981-9e2f-885a7705b066\") " pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.746773 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.746717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn7f\" (UniqueName: \"kubernetes.io/projected/e22e24ba-97a9-42e4-9bf1-16d3aed89805-kube-api-access-lcn7f\") pod \"tuned-mbnkx\" (UID: \"e22e24ba-97a9-42e4-9bf1-16d3aed89805\") " pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.746958 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.746932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9m8\" (UniqueName: \"kubernetes.io/projected/0c8b4111-5914-46e9-928b-0f33f2d1ac7e-kube-api-access-vf9m8\") pod \"multus-bgqcq\" (UID: \"0c8b4111-5914-46e9-928b-0f33f2d1ac7e\") " pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.806486 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.806458    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:25.822361 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.822336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:25.842482 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.842296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-54jpc"
Jul 13 18:49:25.852807 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.852786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2rdtb"
Jul 13 18:49:25.858850 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.858830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgqcq"
Jul 13 18:49:25.865900 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.865869    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:25.873678 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.873659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp"
Jul 13 18:49:25.885403 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.885385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ql9cx"
Jul 13 18:49:25.893194 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.893173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mbnkx"
Jul 13 18:49:25.899888 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:25.899859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pd2tc"
Jul 13 18:49:26.138752 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.138671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:26.138906 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.138839    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:26.138952 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.138908    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:49:27.138893159 +0000 UTC m=+4.039118535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:26.239719 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.239672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:26.239926 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.239875    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:26.240002 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.239952    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret podName:c020b2f3-f25b-4d53-8e98-360ac0d1ede6 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:27.239931173 +0000 UTC m=+4.140156546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret") pod "global-pull-secret-syncer-q4wtd" (UID: "c020b2f3-f25b-4d53-8e98-360ac0d1ede6") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:26.340909 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.340870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:26.341062 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.341032    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:26.341062 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.341056    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:26.341062 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.341066    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xmchq for pod openshift-network-diagnostics/network-check-target-5l5l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:26.341229 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:26.341126    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq podName:9748f76c-1486-4756-90f7-4ab1d24bcc11 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:27.341106911 +0000 UTC m=+4.241332285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmchq" (UniqueName: "kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq") pod "network-check-target-5l5l6" (UID: "9748f76c-1486-4756-90f7-4ab1d24bcc11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:26.541743 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.541709    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22e24ba_97a9_42e4_9bf1_16d3aed89805.slice/crio-3952b2b53bf2a1c91e0c4c14e892cbe4ceb9502b3790990ed2652ef487f240ba WatchSource:0}: Error finding container 3952b2b53bf2a1c91e0c4c14e892cbe4ceb9502b3790990ed2652ef487f240ba: Status 404 returned error can't find the container with id 3952b2b53bf2a1c91e0c4c14e892cbe4ceb9502b3790990ed2652ef487f240ba
Jul 13 18:49:26.543046 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.542981    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e98cef_d1a2_4caf_af2b_22ea55efc5ac.slice/crio-f1fc7fceab729b7ef66b52a7fa97ac876e4448cccce1d2e9da296a512c217c35 WatchSource:0}: Error finding container f1fc7fceab729b7ef66b52a7fa97ac876e4448cccce1d2e9da296a512c217c35: Status 404 returned error can't find the container with id f1fc7fceab729b7ef66b52a7fa97ac876e4448cccce1d2e9da296a512c217c35
Jul 13 18:49:26.543737 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.543346    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8b4111_5914_46e9_928b_0f33f2d1ac7e.slice/crio-f9a61443a6e5b2546d9b99711f9184d3d07a9c96bc8d49a92c9cae01f739933e WatchSource:0}: Error finding container f9a61443a6e5b2546d9b99711f9184d3d07a9c96bc8d49a92c9cae01f739933e: Status 404 returned error can't find the container with id f9a61443a6e5b2546d9b99711f9184d3d07a9c96bc8d49a92c9cae01f739933e
Jul 13 18:49:26.547364 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.547341    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e212d7_c36c_4e94_86b9_c34e88752fce.slice/crio-ffe97a607bc9a6995d9edfe81b475fdc058c24f73d5aacc617798a9aa599d8b7 WatchSource:0}: Error finding container ffe97a607bc9a6995d9edfe81b475fdc058c24f73d5aacc617798a9aa599d8b7: Status 404 returned error can't find the container with id ffe97a607bc9a6995d9edfe81b475fdc058c24f73d5aacc617798a9aa599d8b7
Jul 13 18:49:26.571174 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.571147    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6cb61f_f3c0_4ef3_9a41_3143005c2fbf.slice/crio-847a2bf3b05e8d193675ead6944d29dce475920fd96015eeb711f0fa566d9453 WatchSource:0}: Error finding container 847a2bf3b05e8d193675ead6944d29dce475920fd96015eeb711f0fa566d9453: Status 404 returned error can't find the container with id 847a2bf3b05e8d193675ead6944d29dce475920fd96015eeb711f0fa566d9453
Jul 13 18:49:26.572749 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.572723    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf537d94_8daa_47b7_9c7e_02e3c5e25f0c.slice/crio-3fd36934657442bea6f19a475344cbe97685427d64efec97db442f69b7898baf WatchSource:0}: Error finding container 3fd36934657442bea6f19a475344cbe97685427d64efec97db442f69b7898baf: Status 404 returned error can't find the container with id 3fd36934657442bea6f19a475344cbe97685427d64efec97db442f69b7898baf
Jul 13 18:49:26.573766 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.573706    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f5f16f_b1dc_44be_b416_da33fca52c24.slice/crio-55111d2758ddef2abdcd0800c8e757d19f8e51cef546a7c7f6333a63e95b23eb WatchSource:0}: Error finding container 55111d2758ddef2abdcd0800c8e757d19f8e51cef546a7c7f6333a63e95b23eb: Status 404 returned error can't find the container with id 55111d2758ddef2abdcd0800c8e757d19f8e51cef546a7c7f6333a63e95b23eb
Jul 13 18:49:26.574916 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.574889    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e61056f_7549_4981_9e2f_885a7705b066.slice/crio-97f83c1c89358c6152b8a5887844697b9ccd40fb4cb21d89ecb735bd2343454a WatchSource:0}: Error finding container 97f83c1c89358c6152b8a5887844697b9ccd40fb4cb21d89ecb735bd2343454a: Status 404 returned error can't find the container with id 97f83c1c89358c6152b8a5887844697b9ccd40fb4cb21d89ecb735bd2343454a
Jul 13 18:49:26.575533 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:26.575430    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63f3d9e_26c5_4199_b3f1_402a91a4e5ed.slice/crio-c34ec1ff149cfe667ebf2fda22a8ef9dd50b6919ed29b51eb3623d732c7236d2 WatchSource:0}: Error finding container c34ec1ff149cfe667ebf2fda22a8ef9dd50b6919ed29b51eb3623d732c7236d2: Status 404 returned error can't find the container with id c34ec1ff149cfe667ebf2fda22a8ef9dd50b6919ed29b51eb3623d732c7236d2
Jul 13 18:49:26.584636 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.584611    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:44:26 +0000 UTC" deadline="2027-12-19 02:27:16.644437911 +0000 UTC"
Jul 13 18:49:26.584692 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.584637    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12559h37m50.05980492s"
Jul 13 18:49:26.606139 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.606112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp" event={"ID":"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed","Type":"ContainerStarted","Data":"c34ec1ff149cfe667ebf2fda22a8ef9dd50b6919ed29b51eb3623d732c7236d2"}
Jul 13 18:49:26.612299 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.612265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4qmlg" event={"ID":"f6e212d7-c36c-4e94-86b9-c34e88752fce","Type":"ContainerStarted","Data":"ffe97a607bc9a6995d9edfe81b475fdc058c24f73d5aacc617798a9aa599d8b7"}
Jul 13 18:49:26.617311 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.617288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgqcq" event={"ID":"0c8b4111-5914-46e9-928b-0f33f2d1ac7e","Type":"ContainerStarted","Data":"f9a61443a6e5b2546d9b99711f9184d3d07a9c96bc8d49a92c9cae01f739933e"}
Jul 13 18:49:26.620719 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.620699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerStarted","Data":"f1fc7fceab729b7ef66b52a7fa97ac876e4448cccce1d2e9da296a512c217c35"}
Jul 13 18:49:26.621776 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.621757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pd2tc" event={"ID":"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf","Type":"ContainerStarted","Data":"847a2bf3b05e8d193675ead6944d29dce475920fd96015eeb711f0fa566d9453"}
Jul 13 18:49:26.622855 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.622831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ql9cx" event={"ID":"9e61056f-7549-4981-9e2f-885a7705b066","Type":"ContainerStarted","Data":"97f83c1c89358c6152b8a5887844697b9ccd40fb4cb21d89ecb735bd2343454a"}
Jul 13 18:49:26.624436 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.624122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"3fd36934657442bea6f19a475344cbe97685427d64efec97db442f69b7898baf"}
Jul 13 18:49:26.625190 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.625163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-54jpc" event={"ID":"b0f5f16f-b1dc-44be-b416-da33fca52c24","Type":"ContainerStarted","Data":"55111d2758ddef2abdcd0800c8e757d19f8e51cef546a7c7f6333a63e95b23eb"}
Jul 13 18:49:26.626246 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:26.626228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mbnkx" event={"ID":"e22e24ba-97a9-42e4-9bf1-16d3aed89805","Type":"ContainerStarted","Data":"3952b2b53bf2a1c91e0c4c14e892cbe4ceb9502b3790990ed2652ef487f240ba"}
Jul 13 18:49:27.148343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.148255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:27.148527 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.148429    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:27.148527 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.148514    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:49:29.148491219 +0000 UTC m=+6.048716595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:27.249565 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.248982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:27.249565 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.249133    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:27.249565 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.249198    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret podName:c020b2f3-f25b-4d53-8e98-360ac0d1ede6 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:29.249178773 +0000 UTC m=+6.149404146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret") pod "global-pull-secret-syncer-q4wtd" (UID: "c020b2f3-f25b-4d53-8e98-360ac0d1ede6") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:27.352540 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.352455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:27.352731 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.352659    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:27.352731 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.352688    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:27.352731 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.352703    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xmchq for pod openshift-network-diagnostics/network-check-target-5l5l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:27.352937 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.352777    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq podName:9748f76c-1486-4756-90f7-4ab1d24bcc11 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:29.352756402 +0000 UTC m=+6.252981785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmchq" (UniqueName: "kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq") pod "network-check-target-5l5l6" (UID: "9748f76c-1486-4756-90f7-4ab1d24bcc11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:27.502012 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.501950    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:27.584812 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.584744    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:44:26 +0000 UTC" deadline="2027-12-08 05:44:48.699551006 +0000 UTC"
Jul 13 18:49:27.585323 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.584823    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12298h55m21.114732147s"
Jul 13 18:49:27.601337 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.600394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:27.601337 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.600525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:27.601337 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.600989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:27.601337 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.601106    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:27.601337 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.601166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:27.601337 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:27.601231    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:27.644668 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.644626    2581 generic.go:356] "Generic (PLEG): container finished" podID="65659a82159918e78dc6edc085316252" containerID="1df61c566984b6cbb1fe0de95ce21bc003357e97a9e3a84aa4d1a7e21777237b" exitCode=0
Jul 13 18:49:27.644846 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.644731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal" event={"ID":"65659a82159918e78dc6edc085316252","Type":"ContainerDied","Data":"1df61c566984b6cbb1fe0de95ce21bc003357e97a9e3a84aa4d1a7e21777237b"}
Jul 13 18:49:27.647583 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:27.647536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal" event={"ID":"58f7c83942b70dd267414f6b25e3812d","Type":"ContainerStarted","Data":"7e5e0e0415863f3bb0aae3fabd9057221734fc2a94f3ea933a7e0a6b0f53a7ad"}
Jul 13 18:49:28.659238 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:28.659189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal" event={"ID":"65659a82159918e78dc6edc085316252","Type":"ContainerStarted","Data":"9f5de9a296750425c143c5d00824b9638629a977085733f0bc87fab247c7e996"}
Jul 13 18:49:28.676100 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:28.675143    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal" podStartSLOduration=3.675120266 podStartE2EDuration="3.675120266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:27.682491475 +0000 UTC m=+4.582716873" watchObservedRunningTime="2026-07-13 18:49:28.675120266 +0000 UTC m=+5.575345662"
Jul 13 18:49:28.676100 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:28.675316    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-203.ec2.internal" podStartSLOduration=3.675309464 podStartE2EDuration="3.675309464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:28.674853813 +0000 UTC m=+5.575079209" watchObservedRunningTime="2026-07-13 18:49:28.675309464 +0000 UTC m=+5.575534861"
Jul 13 18:49:29.167972 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:29.167369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:29.167972 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.167531    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:29.167972 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.167597    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:49:33.167577666 +0000 UTC m=+10.067803055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:29.268356 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:29.267733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:29.268356 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.267929    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:29.268356 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.268000    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret podName:c020b2f3-f25b-4d53-8e98-360ac0d1ede6 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:33.267981291 +0000 UTC m=+10.168206665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret") pod "global-pull-secret-syncer-q4wtd" (UID: "c020b2f3-f25b-4d53-8e98-360ac0d1ede6") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:29.368324 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:29.368276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:29.368526 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.368452    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:29.368526 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.368479    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:29.368526 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.368495    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xmchq for pod openshift-network-diagnostics/network-check-target-5l5l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:29.368694 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.368558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq podName:9748f76c-1486-4756-90f7-4ab1d24bcc11 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:33.368537846 +0000 UTC m=+10.268763223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmchq" (UniqueName: "kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq") pod "network-check-target-5l5l6" (UID: "9748f76c-1486-4756-90f7-4ab1d24bcc11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:29.597800 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:29.597716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:29.597993 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.597867    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:29.598274 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:29.598253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:29.598377 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.598356    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:29.598443 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:29.598414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:29.598530 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:29.598508    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:31.597698 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:31.597661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:31.598206 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:31.597793    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:31.598206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:31.597890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:31.598206 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:31.597972    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:31.598206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:31.598037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:31.598206 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:31.598113    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:33.204663 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:33.204623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:33.205107 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.204840    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:33.205107 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.204907    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:49:41.204886144 +0000 UTC m=+18.105111520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:33.305595 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:33.305051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:33.305595 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.305216    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:33.305595 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.305275    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret podName:c020b2f3-f25b-4d53-8e98-360ac0d1ede6 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:41.305261582 +0000 UTC m=+18.205486951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret") pod "global-pull-secret-syncer-q4wtd" (UID: "c020b2f3-f25b-4d53-8e98-360ac0d1ede6") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:33.406343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:33.406307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:33.406536 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.406515    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:33.406536 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.406541    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:33.406698 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.406554    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xmchq for pod openshift-network-diagnostics/network-check-target-5l5l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:33.406698 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.406611    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq podName:9748f76c-1486-4756-90f7-4ab1d24bcc11 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:41.406590749 +0000 UTC m=+18.306816125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmchq" (UniqueName: "kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq") pod "network-check-target-5l5l6" (UID: "9748f76c-1486-4756-90f7-4ab1d24bcc11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:33.604287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:33.604208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:33.604287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:33.604272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:33.604494 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:33.604377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:33.604494 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.604383    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:33.604494 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.604457    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:33.604631 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:33.604530    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:35.598150 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:35.598071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:35.598588 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:35.598071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:35.598588 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:35.598207    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:35.598588 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:35.598078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:35.598588 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:35.598291    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:35.598588 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:35.598327    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:37.598017 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:37.597981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:37.598017 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:37.597999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:37.598515 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:37.597998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:37.598515 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:37.598084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:37.598515 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:37.598190    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:37.598515 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:37.598289    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:39.598055 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:39.598014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:39.598534 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:39.598026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:39.598534 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:39.598143    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:39.598534 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:39.598170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:39.598534 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:39.598228    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:39.598534 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:39.598309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:41.274603 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:41.274577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:41.274953 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.274712    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:41.274953 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.274768    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:49:57.274752569 +0000 UTC m=+34.174977939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:41.375371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:41.375330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:41.375526 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.375463    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:41.375526 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.375515    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret podName:c020b2f3-f25b-4d53-8e98-360ac0d1ede6 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:57.375501951 +0000 UTC m=+34.275727325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret") pod "global-pull-secret-syncer-q4wtd" (UID: "c020b2f3-f25b-4d53-8e98-360ac0d1ede6") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:41.476644 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:41.476568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:41.476821 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.476719    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:41.476821 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.476739    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:41.476821 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.476750    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xmchq for pod openshift-network-diagnostics/network-check-target-5l5l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:41.476980 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.476830    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq podName:9748f76c-1486-4756-90f7-4ab1d24bcc11 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:57.476816724 +0000 UTC m=+34.377042094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmchq" (UniqueName: "kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq") pod "network-check-target-5l5l6" (UID: "9748f76c-1486-4756-90f7-4ab1d24bcc11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:41.598009 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:41.597972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:41.598175 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:41.597983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:41.598175 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.598113    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:41.598175 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:41.597987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:41.598343 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.598184    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:41.598343 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:41.598244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:43.599453 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:43.599160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:43.599453 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:43.599266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:43.599453 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:43.599310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:43.599453 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:43.599322    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:43.599453 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:43.599407    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:43.600166 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:43.599511    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:45.597694 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:45.597657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:45.598233 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:45.597663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:45.598233 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:45.597762    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:45.598233 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:45.597670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:45.598233 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:45.597837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:45.598233 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:45.597908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:46.699145 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.698749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp" event={"ID":"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed","Type":"ContainerStarted","Data":"99731bddb635c716e0cc463b9e01eefe7254bb9a95aefc0978f4a440332446b0"}
Jul 13 18:49:46.700245 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.700221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4qmlg" event={"ID":"f6e212d7-c36c-4e94-86b9-c34e88752fce","Type":"ContainerStarted","Data":"bbbb2684f5aa91e8c3649b970f0e037493945f6a5f1b21fcc0e5ca763b12de14"}
Jul 13 18:49:46.701930 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.701902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgqcq" event={"ID":"0c8b4111-5914-46e9-928b-0f33f2d1ac7e","Type":"ContainerStarted","Data":"3b925f74bc744f38868778fd7f1fcac8337fcca85cfb38f93634a8aefa8ab02e"}
Jul 13 18:49:46.703698 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.703674    2581 generic.go:356] "Generic (PLEG): container finished" podID="96e98cef-d1a2-4caf-af2b-22ea55efc5ac" containerID="693562be9b1add5a5048fe071b082bca870c64a830ee5a04e6a365831bb37dfd" exitCode=0
Jul 13 18:49:46.703778 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.703750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerDied","Data":"693562be9b1add5a5048fe071b082bca870c64a830ee5a04e6a365831bb37dfd"}
Jul 13 18:49:46.705407 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.705338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pd2tc" event={"ID":"2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf","Type":"ContainerStarted","Data":"497287e0fb423a056ba041cab72377869906b9c2f5fe3744d6d36333ef34e15d"}
Jul 13 18:49:46.707090 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.707060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ql9cx" event={"ID":"9e61056f-7549-4981-9e2f-885a7705b066","Type":"ContainerStarted","Data":"d0d8e9a2e10be6a022dea1d48d9e6843ddc4a49c7c91ee1028593a671ff8fca4"}
Jul 13 18:49:46.708633 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.708618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:49:46.708996 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.708975    2581 generic.go:356] "Generic (PLEG): container finished" podID="cf537d94-8daa-47b7-9c7e-02e3c5e25f0c" containerID="bd04c259dab3e5dbd10d48f57e2c4a29b8220200aaeb2b86ed09c0a174ca27d1" exitCode=1
Jul 13 18:49:46.709065 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.709034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"959c6b5224c471373741a7ddc64bda537244114500096ad7e7209070fa39ac23"}
Jul 13 18:49:46.709178 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.709100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerDied","Data":"bd04c259dab3e5dbd10d48f57e2c4a29b8220200aaeb2b86ed09c0a174ca27d1"}
Jul 13 18:49:46.709178 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.709149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"4b16bd1ac6eda05cd8692cbca348e47367d899b57809eb199597e80c9724f6a9"}
Jul 13 18:49:46.710493 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.710470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mbnkx" event={"ID":"e22e24ba-97a9-42e4-9bf1-16d3aed89805","Type":"ContainerStarted","Data":"2e5215af5f9cafff6d53b9ef066ba6e5c9b8b5b77a404f194978c5032dc6a3c8"}
Jul 13 18:49:46.714558 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.714523    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4qmlg" podStartSLOduration=8.083973038 podStartE2EDuration="22.714511883s" totalImagesPullingTime="14.630538845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:46.714493686 +0000 UTC m=+23.614719088" watchObservedRunningTime="2026-07-13 18:49:46.714511883 +0000 UTC m=+23.614737274"
Jul 13 18:49:46.747454 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.747011    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ql9cx" podStartSLOduration=3.285910485 podStartE2EDuration="22.746991603s" totalImagesPullingTime="19.461081118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:46.746931666 +0000 UTC m=+23.647157058" watchObservedRunningTime="2026-07-13 18:49:46.746991603 +0000 UTC m=+23.647216995"
Jul 13 18:49:46.764541 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.763968    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bgqcq" podStartSLOduration=2.891148324 podStartE2EDuration="22.763949795s" totalImagesPullingTime="19.872801471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:46.7637416 +0000 UTC m=+23.663966994" watchObservedRunningTime="2026-07-13 18:49:46.763949795 +0000 UTC m=+23.664175191"
Jul 13 18:49:46.781579 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:46.781529    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mbnkx" podStartSLOduration=3.28410481 podStartE2EDuration="22.781513644s" totalImagesPullingTime="19.497408834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:46.781016087 +0000 UTC m=+23.681241480" watchObservedRunningTime="2026-07-13 18:49:46.781513644 +0000 UTC m=+23.681739037"
Jul 13 18:49:47.556439 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.556283    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 18:49:47.596775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.596689    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T18:49:47.556436926Z","UUID":"304c2ecd-a53a-4785-94c0-0003ca2e876d","Handler":null,"Name":"","Endpoint":""}
Jul 13 18:49:47.597652 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.597627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:47.597829 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:47.597762    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:47.597992 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.597976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:47.598113 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:47.598074    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:47.598228 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.598163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:47.598282 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:47.598246    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:47.598501 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.598473    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 18:49:47.598571 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.598512    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 18:49:47.716675 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.716599    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:49:47.717154 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.717081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"50c9f480f0336c9a7fdb538c5b16381ec21106c1bf763e0836e601e32a04dbe2"}
Jul 13 18:49:47.717226 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.717153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"6f1cb1c2ea1a209348e113a0a10c16d96112790397be4acaa2392cb842620e5c"}
Jul 13 18:49:47.717226 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.717201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"93ad8ebf849f89b49aa2b49bc2b839260bed81c34a34ae9c1c35ea7a5ab97083"}
Jul 13 18:49:47.718641 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.718616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-54jpc" event={"ID":"b0f5f16f-b1dc-44be-b416-da33fca52c24","Type":"ContainerStarted","Data":"05dd0f0a8d0a39fecf0313c789ff3dc16f203c255a3935b097d0bf46f4baae0b"}
Jul 13 18:49:47.720801 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.720763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp" event={"ID":"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed","Type":"ContainerStarted","Data":"5d653b7f39b9bdf5cef799adbcdd58f9240e297e6dc026eae5733bfdad4cddb9"}
Jul 13 18:49:47.736405 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.736365    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pd2tc" podStartSLOduration=4.268245476 podStartE2EDuration="23.73635017s" totalImagesPullingTime="19.468104694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:46.795632804 +0000 UTC m=+23.695858193" watchObservedRunningTime="2026-07-13 18:49:47.73635017 +0000 UTC m=+24.636575564"
Jul 13 18:49:47.736521 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:47.736497    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-54jpc" podStartSLOduration=4.299639795 podStartE2EDuration="23.736490581s" totalImagesPullingTime="19.436850786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:47.736195619 +0000 UTC m=+24.636421011" watchObservedRunningTime="2026-07-13 18:49:47.736490581 +0000 UTC m=+24.636715973"
Jul 13 18:49:49.598823 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:49.598371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:49.598823 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:49.598501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:49.598823 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:49.598502    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:49.598823 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:49.598380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:49.598823 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:49.598601    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:49.598823 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:49.598687    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:49.727160 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:49.727124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp" event={"ID":"b63f3d9e-26c5-4199-b3f1-402a91a4e5ed","Type":"ContainerStarted","Data":"76364d76caaf0b54cdad3311bb0df5ef42c98fe1d17f21bbbd225123f7881ea1"}
Jul 13 18:49:49.730448 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:49.730415    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:49:49.730958 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:49.730925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"a39c659212be1569d76e4c20bb9d823e7954398f6015270eec30b46e81ee9b35"}
Jul 13 18:49:49.745674 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:49.745632    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmtjp" podStartSLOduration=8.821581744 podStartE2EDuration="25.745619791s" totalImagesPullingTime="16.924038047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:49.745452901 +0000 UTC m=+26.645678293" watchObservedRunningTime="2026-07-13 18:49:49.745619791 +0000 UTC m=+26.645845183"
Jul 13 18:49:51.598494 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.598321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:51.599071 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.598328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:51.599071 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:51.598573    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:51.599071 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:51.598672    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:51.599071 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.598338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:51.599071 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:51.598819    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:51.688594 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.688570    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:51.689166 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.689149    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:51.738132 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.738107    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:49:51.738475 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.738448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"a946337f096d4d2dde3540e6244423e7c2746de45303cd0b684cca46faa86f81"}
Jul 13 18:49:51.738724 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.738707    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:51.738809 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.738783    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:51.738871 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.738847    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:51.738917 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.738893    2581 scope.go:117] "RemoveContainer" containerID="bd04c259dab3e5dbd10d48f57e2c4a29b8220200aaeb2b86ed09c0a174ca27d1"
Jul 13 18:49:51.740552 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.740527    2581 generic.go:356] "Generic (PLEG): container finished" podID="96e98cef-d1a2-4caf-af2b-22ea55efc5ac" containerID="87ebd84f7c0b357417623cc22e3525f150b3a55d5c53112d84294aa828a503a3" exitCode=0
Jul 13 18:49:51.740834 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.740609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerDied","Data":"87ebd84f7c0b357417623cc22e3525f150b3a55d5c53112d84294aa828a503a3"}
Jul 13 18:49:51.740962 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.740931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:51.742209 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.742113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4qmlg"
Jul 13 18:49:51.756079 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.756058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:51.756268 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:51.756254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:49:52.747076 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:52.746946    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:49:52.747519 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:52.747496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" event={"ID":"cf537d94-8daa-47b7-9c7e-02e3c5e25f0c","Type":"ContainerStarted","Data":"50afe396943ccdf6cc1d15087864d4d29981240245f09e65e6cf3e5e5a5ce189"}
Jul 13 18:49:52.774723 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:52.774673    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k" podStartSLOduration=9.254813796 podStartE2EDuration="28.774656893s" totalImagesPullingTime="19.519843097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:52.774478819 +0000 UTC m=+29.674704211" watchObservedRunningTime="2026-07-13 18:49:52.774656893 +0000 UTC m=+29.674882285"
Jul 13 18:49:53.598514 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.598482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:53.598514 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.598513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:53.598711 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:53.598610    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:53.598711 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:53.598635    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:53.598711 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.598669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:53.598838 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:53.598737    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:53.615579 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.615549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q4wtd"]
Jul 13 18:49:53.618735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.618709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5l5l6"]
Jul 13 18:49:53.621131 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.621108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-45fpx"]
Jul 13 18:49:53.752643 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.752615    2581 generic.go:356] "Generic (PLEG): container finished" podID="96e98cef-d1a2-4caf-af2b-22ea55efc5ac" containerID="e74fafc2565ebf2092a2abff57311282c537aa78e98052dd15a76004572ab348" exitCode=0
Jul 13 18:49:53.753024 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.752683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:53.753024 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.752717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerDied","Data":"e74fafc2565ebf2092a2abff57311282c537aa78e98052dd15a76004572ab348"}
Jul 13 18:49:53.753024 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:53.752967    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:53.753143 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.753072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:53.753180 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:53.753154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:53.753180 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:53.753176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:53.753282 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:53.753266    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:54.757147 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:54.757116    2581 generic.go:356] "Generic (PLEG): container finished" podID="96e98cef-d1a2-4caf-af2b-22ea55efc5ac" containerID="3d30de01ba694587463eab367d06f0d27aa86c062c1a34e3fd324158c6519570" exitCode=0
Jul 13 18:49:54.757631 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:54.757153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerDied","Data":"3d30de01ba694587463eab367d06f0d27aa86c062c1a34e3fd324158c6519570"}
Jul 13 18:49:55.597475 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:55.597444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:55.597625 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:55.597480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:55.597625 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:55.597446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:55.597625 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:55.597579    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:55.597767 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:55.597641    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:55.597767 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:55.597738    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:57.283213 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:57.283182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:57.283968 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.283319    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:57.283968 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.283391    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:50:29.283375323 +0000 UTC m=+66.183600692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:57.384444 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:57.384412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:57.384605 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.384508    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:57.384605 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.384556    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret podName:c020b2f3-f25b-4d53-8e98-360ac0d1ede6 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:29.384543464 +0000 UTC m=+66.284768835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret") pod "global-pull-secret-syncer-q4wtd" (UID: "c020b2f3-f25b-4d53-8e98-360ac0d1ede6") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:57.484886 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:57.484850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:57.485060 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.485021    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:57.485060 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.485043    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:57.485060 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.485053    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xmchq for pod openshift-network-diagnostics/network-check-target-5l5l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:57.485253 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.485118    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq podName:9748f76c-1486-4756-90f7-4ab1d24bcc11 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:29.485098223 +0000 UTC m=+66.385323614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xmchq" (UniqueName: "kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq") pod "network-check-target-5l5l6" (UID: "9748f76c-1486-4756-90f7-4ab1d24bcc11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:57.597551 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:57.597472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:57.597551 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:57.597506    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:57.597721 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:57.597473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:57.597721 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.597616    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45fpx" podUID="1e985b1a-0340-49e6-be50-39cf1ae8fe7c"
Jul 13 18:49:57.597721 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.597711    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4wtd" podUID="c020b2f3-f25b-4d53-8e98-360ac0d1ede6"
Jul 13 18:49:57.597896 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:57.597801    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5l5l6" podUID="9748f76c-1486-4756-90f7-4ab1d24bcc11"
Jul 13 18:49:58.892477 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.892442    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-203.ec2.internal" event="NodeReady"
Jul 13 18:49:58.892900 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.892561    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 18:49:58.927801 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.927750    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"]
Jul 13 18:49:58.946268 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.946242    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"]
Jul 13 18:49:58.946401 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.946373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:58.949112 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.948933    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 18:49:58.949112 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.949104    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 18:49:58.949385 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.949365    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 13 18:49:58.949845 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.949826    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 18:49:58.961149 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.961129    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"]
Jul 13 18:49:58.961266 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.961249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:58.963895 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.963876    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 13 18:49:58.963895 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.963890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 13 18:49:58.964034 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.963914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 13 18:49:58.964034 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.963925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 13 18:49:58.973013 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.972991    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"]
Jul 13 18:49:58.973109 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.973099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:58.975096 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.975079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-89v74\""
Jul 13 18:49:58.975209 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.975139    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 13 18:49:58.993510 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.993493    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"]
Jul 13 18:49:58.993611 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.993521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"]
Jul 13 18:49:58.993611 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.993535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"]
Jul 13 18:49:58.993611 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.993554    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wnkpx"]
Jul 13 18:49:58.993770 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.993618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:58.996034 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.996020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 18:49:58.996111 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.996036    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 18:49:58.996179 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.996164    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p56b4\""
Jul 13 18:49:58.996232 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:58.996209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 18:49:59.010009 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.009987    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7f688"]
Jul 13 18:49:59.010215 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.010188    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 18:49:59.010285 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.010275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.012467 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.012450    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 18:49:59.012564 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.012469    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 18:49:59.012773 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.012757    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bq5zm\""
Jul 13 18:49:59.012773 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.012771    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 18:49:59.030514 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.030481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"]
Jul 13 18:49:59.030514 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.030508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wnkpx"]
Jul 13 18:49:59.030664 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.030533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7f688"]
Jul 13 18:49:59.030664 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.030575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.032497 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.032484    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-88cff\""
Jul 13 18:49:59.033286 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.033273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 18:49:59.033340 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.033297    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 18:49:59.094211 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-image-registry-private-configuration\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094211 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cb09706-4278-4a46-953f-d86441a40fe7-ca-trust-extracted\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094211 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-bound-sa-token\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d722ad-8eec-4a87-abe2-38136057f087-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.094378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czt48\" (UniqueName: \"kubernetes.io/projected/d3d722ad-8eec-4a87-abe2-38136057f087-kube-api-access-czt48\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.094378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.094378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87cfa553-bf84-4389-9b3a-7328ef2a7268-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499\" (UID: \"87cfa553-bf84-4389-9b3a-7328ef2a7268\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:59.094378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt6l8\" (UniqueName: \"kubernetes.io/projected/87cfa553-bf84-4389-9b3a-7328ef2a7268-kube-api-access-xt6l8\") pod \"managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499\" (UID: \"87cfa553-bf84-4389-9b3a-7328ef2a7268\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:59.094378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df5ee2c4-9ea0-4224-86b3-a818887f5a13-tmp\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.094378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/df5ee2c4-9ea0-4224-86b3-a818887f5a13-klusterlet-config\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-registry-certificates\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-installation-pull-secrets\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfr5z\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-kube-api-access-cfr5z\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrsx\" (UniqueName: \"kubernetes.io/projected/df5ee2c4-9ea0-4224-86b3-a818887f5a13-kube-api-access-qzrsx\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-trusted-ca\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-ca\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.094655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.094571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-hub\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.122817 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.122776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:49:59.195697 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrbz\" (UniqueName: \"kubernetes.io/projected/b19da82d-b0dc-40ac-ad54-b8acae858b1a-kube-api-access-kbrbz\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.195831 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-trusted-ca\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.195831 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-ca\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.195831 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-hub\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.195962 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.195962 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-image-registry-private-configuration\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.195962 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cb09706-4278-4a46-953f-d86441a40fe7-ca-trust-extracted\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.196113 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.195970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-bound-sa-token\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.196113 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d722ad-8eec-4a87-abe2-38136057f087-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.196113 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czt48\" (UniqueName: \"kubernetes.io/projected/d3d722ad-8eec-4a87-abe2-38136057f087-kube-api-access-czt48\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.196113 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.196113 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87cfa553-bf84-4389-9b3a-7328ef2a7268-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499\" (UID: \"87cfa553-bf84-4389-9b3a-7328ef2a7268\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:59.196321 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt6l8\" (UniqueName: \"kubernetes.io/projected/87cfa553-bf84-4389-9b3a-7328ef2a7268-kube-api-access-xt6l8\") pod \"managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499\" (UID: \"87cfa553-bf84-4389-9b3a-7328ef2a7268\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:59.196321 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bea1125-1479-43d3-a72b-411ddb57cd13-cert\") pod \"ingress-canary-wnkpx\" (UID: \"1bea1125-1479-43d3-a72b-411ddb57cd13\") " pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.196408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhg6\" (UniqueName: \"kubernetes.io/projected/1bea1125-1479-43d3-a72b-411ddb57cd13-kube-api-access-fmhg6\") pod \"ingress-canary-wnkpx\" (UID: \"1bea1125-1479-43d3-a72b-411ddb57cd13\") " pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.196459 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df5ee2c4-9ea0-4224-86b3-a818887f5a13-tmp\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.196459 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/df5ee2c4-9ea0-4224-86b3-a818887f5a13-klusterlet-config\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.196561 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-registry-certificates\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.196561 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-installation-pull-secrets\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.196657 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19da82d-b0dc-40ac-ad54-b8acae858b1a-tmp-dir\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.196657 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.196657 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b19da82d-b0dc-40ac-ad54-b8acae858b1a-config-volume\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.196817 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.196817 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfr5z\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-kube-api-access-cfr5z\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.196817 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrsx\" (UniqueName: \"kubernetes.io/projected/df5ee2c4-9ea0-4224-86b3-a818887f5a13-kube-api-access-qzrsx\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.196817 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d722ad-8eec-4a87-abe2-38136057f087-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.197003 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-trusted-ca\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.197105 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.197085    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:49:59.197161 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.197107    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:49:59.197211 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.197161    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:59.697142786 +0000 UTC m=+36.597368173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:49:59.197274 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.197241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-registry-certificates\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.197326 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.196214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cb09706-4278-4a46-953f-d86441a40fe7-ca-trust-extracted\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.200595 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.200561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-installation-pull-secrets\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.200708 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.200619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.200708 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.200630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-ca\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.200708 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.200640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-image-registry-private-configuration\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.200874 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.200720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-hub\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.200874 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.200823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87cfa553-bf84-4389-9b3a-7328ef2a7268-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499\" (UID: \"87cfa553-bf84-4389-9b3a-7328ef2a7268\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:59.202480 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.202463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3d722ad-8eec-4a87-abe2-38136057f087-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.203867 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.203847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df5ee2c4-9ea0-4224-86b3-a818887f5a13-tmp\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.206364 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.206335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czt48\" (UniqueName: \"kubernetes.io/projected/d3d722ad-8eec-4a87-abe2-38136057f087-kube-api-access-czt48\") pod \"cluster-proxy-proxy-agent-6b78dcd44c-vpc9h\" (UID: \"d3d722ad-8eec-4a87-abe2-38136057f087\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.206488 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.206340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt6l8\" (UniqueName: \"kubernetes.io/projected/87cfa553-bf84-4389-9b3a-7328ef2a7268-kube-api-access-xt6l8\") pod \"managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499\" (UID: \"87cfa553-bf84-4389-9b3a-7328ef2a7268\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:59.206488 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.206416    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrsx\" (UniqueName: \"kubernetes.io/projected/df5ee2c4-9ea0-4224-86b3-a818887f5a13-kube-api-access-qzrsx\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.206600 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.206580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/df5ee2c4-9ea0-4224-86b3-a818887f5a13-klusterlet-config\") pod \"klusterlet-addon-workmgr-756f654d98-kwkbk\" (UID: \"df5ee2c4-9ea0-4224-86b3-a818887f5a13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.207038 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.207020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-bound-sa-token\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.207100 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.207045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfr5z\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-kube-api-access-cfr5z\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.221381 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.221352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:49:59.257014 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.256992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:49:59.277699 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.277679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:49:59.284097 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.284080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"
Jul 13 18:49:59.297224 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19da82d-b0dc-40ac-ad54-b8acae858b1a-tmp-dir\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.297334 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b19da82d-b0dc-40ac-ad54-b8acae858b1a-config-volume\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.297334 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrbz\" (UniqueName: \"kubernetes.io/projected/b19da82d-b0dc-40ac-ad54-b8acae858b1a-kube-api-access-kbrbz\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.297334 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.297438 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bea1125-1479-43d3-a72b-411ddb57cd13-cert\") pod \"ingress-canary-wnkpx\" (UID: \"1bea1125-1479-43d3-a72b-411ddb57cd13\") " pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.297438 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhg6\" (UniqueName: \"kubernetes.io/projected/1bea1125-1479-43d3-a72b-411ddb57cd13-kube-api-access-fmhg6\") pod \"ingress-canary-wnkpx\" (UID: \"1bea1125-1479-43d3-a72b-411ddb57cd13\") " pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.297507 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.297471    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:49:59.297547 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.297533    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:49:59.797513713 +0000 UTC m=+36.697739089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:49:59.297666 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19da82d-b0dc-40ac-ad54-b8acae858b1a-tmp-dir\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.298035 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.297936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b19da82d-b0dc-40ac-ad54-b8acae858b1a-config-volume\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.299910 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.299893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bea1125-1479-43d3-a72b-411ddb57cd13-cert\") pod \"ingress-canary-wnkpx\" (UID: \"1bea1125-1479-43d3-a72b-411ddb57cd13\") " pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.307921 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.306043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhg6\" (UniqueName: \"kubernetes.io/projected/1bea1125-1479-43d3-a72b-411ddb57cd13-kube-api-access-fmhg6\") pod \"ingress-canary-wnkpx\" (UID: \"1bea1125-1479-43d3-a72b-411ddb57cd13\") " pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.307921 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.306320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrbz\" (UniqueName: \"kubernetes.io/projected/b19da82d-b0dc-40ac-ad54-b8acae858b1a-kube-api-access-kbrbz\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.318671 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.318649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wnkpx"
Jul 13 18:49:59.445893 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.445858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"]
Jul 13 18:49:59.451179 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:59.451123    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5ee2c4_9ea0_4224_86b3_a818887f5a13.slice/crio-a0fc95642185c8a443f3a187130cfbf33b756e8ed43af2f7fd66dfb4749954d5 WatchSource:0}: Error finding container a0fc95642185c8a443f3a187130cfbf33b756e8ed43af2f7fd66dfb4749954d5: Status 404 returned error can't find the container with id a0fc95642185c8a443f3a187130cfbf33b756e8ed43af2f7fd66dfb4749954d5
Jul 13 18:49:59.478980 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.478951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"]
Jul 13 18:49:59.502724 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.502688    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499"]
Jul 13 18:49:59.514766 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.514738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wnkpx"]
Jul 13 18:49:59.518868 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:59.518844    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bea1125_1479_43d3_a72b_411ddb57cd13.slice/crio-53fbefd8ed78cdc78daf50f6166c341a9a9653baddca8e84fce6971978529810 WatchSource:0}: Error finding container 53fbefd8ed78cdc78daf50f6166c341a9a9653baddca8e84fce6971978529810: Status 404 returned error can't find the container with id 53fbefd8ed78cdc78daf50f6166c341a9a9653baddca8e84fce6971978529810
Jul 13 18:49:59.519209 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:59.519190    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87cfa553_bf84_4389_9b3a_7328ef2a7268.slice/crio-21135e41439385851b25c788b3b9cd2050d4fe6f41a047ba7bbc8fa0ed3bccb8 WatchSource:0}: Error finding container 21135e41439385851b25c788b3b9cd2050d4fe6f41a047ba7bbc8fa0ed3bccb8: Status 404 returned error can't find the container with id 21135e41439385851b25c788b3b9cd2050d4fe6f41a047ba7bbc8fa0ed3bccb8
Jul 13 18:49:59.519843 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:49:59.519728    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3d722ad_8eec_4a87_abe2_38136057f087.slice/crio-4f7930db80cac74987ca5b7d2838424172e7d0c172acb011f490a79a4bd95dc2 WatchSource:0}: Error finding container 4f7930db80cac74987ca5b7d2838424172e7d0c172acb011f490a79a4bd95dc2: Status 404 returned error can't find the container with id 4f7930db80cac74987ca5b7d2838424172e7d0c172acb011f490a79a4bd95dc2
Jul 13 18:49:59.598195 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.598163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:49:59.598289 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.598209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:49:59.598289 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.598163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:49:59.600684 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.600668    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dz6tv\""
Jul 13 18:49:59.600995 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.600978    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gsq9l\""
Jul 13 18:49:59.601277 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.601239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:49:59.601277 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.601261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:49:59.601419 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.601247    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:49:59.601419 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.601239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:49:59.700869 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.700831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:49:59.701015 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.700938    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:49:59.701015 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.700950    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:49:59.701015 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.701000    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:00.700985852 +0000 UTC m=+37.601211222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:49:59.772395 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.772355    2581 generic.go:356] "Generic (PLEG): container finished" podID="96e98cef-d1a2-4caf-af2b-22ea55efc5ac" containerID="559a9496276790fe7f2bacd0ad91dc1bc3e47385f442eb4b407d99c4370e102b" exitCode=0
Jul 13 18:49:59.772536 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.772431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerDied","Data":"559a9496276790fe7f2bacd0ad91dc1bc3e47385f442eb4b407d99c4370e102b"}
Jul 13 18:49:59.773613 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.773594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" event={"ID":"d3d722ad-8eec-4a87-abe2-38136057f087","Type":"ContainerStarted","Data":"4f7930db80cac74987ca5b7d2838424172e7d0c172acb011f490a79a4bd95dc2"}
Jul 13 18:49:59.774679 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.774660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wnkpx" event={"ID":"1bea1125-1479-43d3-a72b-411ddb57cd13","Type":"ContainerStarted","Data":"53fbefd8ed78cdc78daf50f6166c341a9a9653baddca8e84fce6971978529810"}
Jul 13 18:49:59.775680 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.775657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk" event={"ID":"df5ee2c4-9ea0-4224-86b3-a818887f5a13","Type":"ContainerStarted","Data":"a0fc95642185c8a443f3a187130cfbf33b756e8ed43af2f7fd66dfb4749954d5"}
Jul 13 18:49:59.776631 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.776592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499" event={"ID":"87cfa553-bf84-4389-9b3a-7328ef2a7268","Type":"ContainerStarted","Data":"21135e41439385851b25c788b3b9cd2050d4fe6f41a047ba7bbc8fa0ed3bccb8"}
Jul 13 18:49:59.801340 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:49:59.801300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:49:59.801488 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.801423    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:49:59.801488 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:49:59.801479    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:00.801464583 +0000 UTC m=+37.701689958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:00.710431 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:00.709535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:50:00.710431 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:00.710007    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:00.710431 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:00.710028    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:50:00.710431 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:00.710086    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:02.71006781 +0000 UTC m=+39.610293185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:50:00.788243 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:00.787547    2581 generic.go:356] "Generic (PLEG): container finished" podID="96e98cef-d1a2-4caf-af2b-22ea55efc5ac" containerID="17e127960f8416356835f9947eacdbb25c7d97baca9b5b3e9a02a99eddfe5d79" exitCode=0
Jul 13 18:50:00.788243 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:00.787623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerDied","Data":"17e127960f8416356835f9947eacdbb25c7d97baca9b5b3e9a02a99eddfe5d79"}
Jul 13 18:50:00.811080 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:00.811040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:50:00.811266 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:00.811249    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:00.811351 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:00.811308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:02.811290388 +0000 UTC m=+39.711515764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:01.798456 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:01.798421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" event={"ID":"96e98cef-d1a2-4caf-af2b-22ea55efc5ac","Type":"ContainerStarted","Data":"4b354237423515003996e4d3bef8a22b568e6bb196c45d737cba44da4262c792"}
Jul 13 18:50:01.824474 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:01.824404    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2rdtb" podStartSLOduration=7.37004904 podStartE2EDuration="37.824356505s" totalImagesPullingTime="30.454307465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:01.823437041 +0000 UTC m=+38.723662432" watchObservedRunningTime="2026-07-13 18:50:01.824356505 +0000 UTC m=+38.724581900"
Jul 13 18:50:01.948561 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:01.948526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:02.048435 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:02.048391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:02.346689 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:02.346652    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:02.424762 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:02.424730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:02.524941 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:02.524897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:02.728974 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:02.728899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:50:02.729131 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:02.729080    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:02.729131 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:02.729107    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:50:02.729229 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:02.729193    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:06.729169262 +0000 UTC m=+43.629394654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:50:02.829559 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:02.829521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:50:02.830271 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:02.830250    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:02.830351 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:02.830318    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:06.830296929 +0000 UTC m=+43.730522300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:03.368334 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:03.368218    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:03.471593 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:03.471511    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:04.296613 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:04.296581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:04.476080 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:04.476011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:06.389473 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:06.389367    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:06.764829 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:06.764770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:50:06.765008 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:06.764941    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:06.765008 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:06.764960    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:50:06.765113 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:06.765020    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:14.765006536 +0000 UTC m=+51.665231906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:50:06.865664 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:06.865625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:50:06.865878 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:06.865823    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:06.865952 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:06.865894    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:14.865875857 +0000 UTC m=+51.766101230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:08.380325 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.380251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:08.435527 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.435495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:08.819462 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.819429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wnkpx" event={"ID":"1bea1125-1479-43d3-a72b-411ddb57cd13","Type":"ContainerStarted","Data":"aab377256be095a0ed15f3f35ae8c612badc752b34e5f392f07e2f43887029b1"}
Jul 13 18:50:08.821532 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.821504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk" event={"ID":"df5ee2c4-9ea0-4224-86b3-a818887f5a13","Type":"ContainerStarted","Data":"8c4a1d60e379514fc8e20570ce4b1f6bf47aa1ab292b30cee63a628bcfc0e8b4"}
Jul 13 18:50:08.821663 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.821645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:50:08.823516 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.823490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499" event={"ID":"87cfa553-bf84-4389-9b3a-7328ef2a7268","Type":"ContainerStarted","Data":"cab7593cb34fe23c6e77dd9664f62dfbb89437cef4b8c8ec4aeaee44edae5f81"}
Jul 13 18:50:08.823656 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.823611    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk"
Jul 13 18:50:08.825719 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.825698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" event={"ID":"d3d722ad-8eec-4a87-abe2-38136057f087","Type":"ContainerStarted","Data":"3e3db4d12eb560093312bec6deaa64f2ce4674a74b025ac45654cca0c3849fe7"}
Jul 13 18:50:08.825825 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.825730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" event={"ID":"d3d722ad-8eec-4a87-abe2-38136057f087","Type":"ContainerStarted","Data":"64c86861c0e39970776d8447dd275d3309746a0578001b9cbd0370dc18aa58d6"}
Jul 13 18:50:08.825825 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.825748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" event={"ID":"d3d722ad-8eec-4a87-abe2-38136057f087","Type":"ContainerStarted","Data":"4785aa7cb4a3dac7d17e7b2b47177700bfbc1fabe184d0494ad7185a6c2017a9"}
Jul 13 18:50:08.835844 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.835804    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wnkpx" podStartSLOduration=2.576184025 podStartE2EDuration="10.835770559s" totalImagesPullingTime="8.259586534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:08.834942314 +0000 UTC m=+45.735167703" watchObservedRunningTime="2026-07-13 18:50:08.835770559 +0000 UTC m=+45.735995945"
Jul 13 18:50:08.852302 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.852249    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-756f654d98-kwkbk" podStartSLOduration=26.525473493 podStartE2EDuration="34.852238303s" totalImagesPullingTime="8.32676481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:08.851152052 +0000 UTC m=+45.751377445" watchObservedRunningTime="2026-07-13 18:50:08.852238303 +0000 UTC m=+45.752463694"
Jul 13 18:50:08.877715 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:08.877666    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" podStartSLOduration=26.605610041 podStartE2EDuration="34.877651929s" totalImagesPullingTime="8.272041888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:08.876607958 +0000 UTC m=+45.776833350" watchObservedRunningTime="2026-07-13 18:50:08.877651929 +0000 UTC m=+45.777877320"
Jul 13 18:50:14.823530 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:14.823491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:50:14.824026 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:14.823695    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:14.824026 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:14.823718    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:50:14.824026 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:14.823811    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:30.823767248 +0000 UTC m=+67.723992640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:50:14.882115 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:14.882069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:14.924621 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:14.924594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:50:14.924743 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:14.924706    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:14.924778 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:14.924762    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:30.924745105 +0000 UTC m=+67.824970475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:14.959467 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:14.959432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:15.067738 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:15.067698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:15.161592 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:15.161495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:15.261210 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:15.261171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:15.875881 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:15.875848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:15.972860 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:15.972834    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:16.046158 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:16.046120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:16.287378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:16.287349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:16.343026 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:16.342992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:18.554614 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:18.554580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:18.653290 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:18.653251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:21.016128 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:21.016085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:23.769971 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:23.769938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2s8k"
Jul 13 18:50:23.798347 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:23.798299    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d9cbc5fd6-2r499" podStartSLOduration=41.539359481 podStartE2EDuration="49.798284916s" totalImagesPullingTime="8.258925435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:08.892978238 +0000 UTC m=+45.793203631" watchObservedRunningTime="2026-07-13 18:50:23.798284916 +0000 UTC m=+60.698510310"
Jul 13 18:50:23.929661 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:23.929622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:23.998688 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:23.998657    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:29.331361 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.331321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:50:29.333767 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.333742    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:50:29.341541 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:29.341519    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 13 18:50:29.341606 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:29.341585    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs podName:1e985b1a-0340-49e6-be50-39cf1ae8fe7c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:33.34157066 +0000 UTC m=+130.241796031 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs") pod "network-metrics-daemon-45fpx" (UID: "1e985b1a-0340-49e6-be50-39cf1ae8fe7c") : secret "metrics-daemon-secret" not found
Jul 13 18:50:29.432625 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.432591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:50:29.435049 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.435032    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:50:29.447138 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.447107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c020b2f3-f25b-4d53-8e98-360ac0d1ede6-original-pull-secret\") pod \"global-pull-secret-syncer-q4wtd\" (UID: \"c020b2f3-f25b-4d53-8e98-360ac0d1ede6\") " pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:50:29.533711 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.533667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:50:29.536226 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.536206    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:50:29.546775 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.546754    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:50:29.557782 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.557763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmchq\" (UniqueName: \"kubernetes.io/projected/9748f76c-1486-4756-90f7-4ab1d24bcc11-kube-api-access-xmchq\") pod \"network-check-target-5l5l6\" (UID: \"9748f76c-1486-4756-90f7-4ab1d24bcc11\") " pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:50:29.614987 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.614922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4wtd"
Jul 13 18:50:29.631473 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.631450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dz6tv\""
Jul 13 18:50:29.639629 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.639608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:50:29.735186 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.735145    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:29.756321 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.756144    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q4wtd"]
Jul 13 18:50:29.758499 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:50:29.758477    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc020b2f3_f25b_4d53_8e98_360ac0d1ede6.slice/crio-c624ccc45874b0dcbd59519219139d8fa1d70906fb054597466d2dfab8085b11 WatchSource:0}: Error finding container c624ccc45874b0dcbd59519219139d8fa1d70906fb054597466d2dfab8085b11: Status 404 returned error can't find the container with id c624ccc45874b0dcbd59519219139d8fa1d70906fb054597466d2dfab8085b11
Jul 13 18:50:29.779136 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.779023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5l5l6"]
Jul 13 18:50:29.781639 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:50:29.781617    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9748f76c_1486_4756_90f7_4ab1d24bcc11.slice/crio-c07511bafa4c702cd9771341c4e4b973be5642927f4ecbb158a18a54b9f8dbe9 WatchSource:0}: Error finding container c07511bafa4c702cd9771341c4e4b973be5642927f4ecbb158a18a54b9f8dbe9: Status 404 returned error can't find the container with id c07511bafa4c702cd9771341c4e4b973be5642927f4ecbb158a18a54b9f8dbe9
Jul 13 18:50:29.889270 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.889194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q4wtd" event={"ID":"c020b2f3-f25b-4d53-8e98-360ac0d1ede6","Type":"ContainerStarted","Data":"c624ccc45874b0dcbd59519219139d8fa1d70906fb054597466d2dfab8085b11"}
Jul 13 18:50:29.890243 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:29.890222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5l5l6" event={"ID":"9748f76c-1486-4756-90f7-4ab1d24bcc11","Type":"ContainerStarted","Data":"c07511bafa4c702cd9771341c4e4b973be5642927f4ecbb158a18a54b9f8dbe9"}
Jul 13 18:50:30.845414 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:30.845373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:50:30.845818 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:30.845550    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:30.845818 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:30.845573    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:50:30.845818 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:30.845642    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:02.845622769 +0000 UTC m=+99.745848153 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:50:30.946395 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:30.946362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:50:30.946595 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:30.946561    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:30.946660 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:30.946622    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:02.946603532 +0000 UTC m=+99.846828904 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:33.904820 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:33.904767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5l5l6" event={"ID":"9748f76c-1486-4756-90f7-4ab1d24bcc11","Type":"ContainerStarted","Data":"840a260c16aa6a46d59a49a41a529110c69b1b2fd49f4c6897a47994080ffb27"}
Jul 13 18:50:33.905278 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:33.904966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:50:33.924666 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:33.924612    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5l5l6" podStartSLOduration=66.763254757 podStartE2EDuration="1m9.924594496s" totalImagesPullingTime="3.161339739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.924359807 +0000 UTC m=+70.824585200" watchObservedRunningTime="2026-07-13 18:50:33.924594496 +0000 UTC m=+70.824819889"
Jul 13 18:50:34.909092 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:34.909057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q4wtd" event={"ID":"c020b2f3-f25b-4d53-8e98-360ac0d1ede6","Type":"ContainerStarted","Data":"63b76e8b9a522ac4f6a378f59046551dfc7b5c27ce535ea2b2f322bf06eb8aa4"}
Jul 13 18:50:34.925890 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:34.925832    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q4wtd" podStartSLOduration=66.274306818 podStartE2EDuration="1m10.925813399s" totalImagesPullingTime="4.651506581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:34.925431166 +0000 UTC m=+71.825656555" watchObservedRunningTime="2026-07-13 18:50:34.925813399 +0000 UTC m=+71.826038788"
Jul 13 18:50:38.393483 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:38.393442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:38.493602 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:38.493570    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:39.265452 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:39.265421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:39.363938 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:39.363907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:39.465635 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:39.465593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:40.342740 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:40.342695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:40.436662 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:40.436630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:40.513940 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:40.513904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:40.574739 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:40.574706    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:43.265010 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:43.264980    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:52.062416 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:52.062371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:52.160951 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:52.160914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.484431 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:54.484388    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.578724 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:54.578690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:55.371665 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:55.371633    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:56.571951 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:56.571919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:56.679309 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:56.679275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:56.782961 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:56.782929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:57.392822 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:57.392774    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:57.500716 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:57.500682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:59.703009 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.702973    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7kmgq"]
Jul 13 18:50:59.707413 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.707398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.710179 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.710158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 18:50:59.710306 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.710224    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c5j2r\""
Jul 13 18:50:59.710432 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.710417    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 18:50:59.714368 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.714351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 18:50:59.714440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.714384    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 18:50:59.729520 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.729495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kmgq"]
Jul 13 18:50:59.789596 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.789561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:59.859672 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.859601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grt8j\" (UniqueName: \"kubernetes.io/projected/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-api-access-grt8j\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.859901 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.859694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.859901 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.859727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.859901 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.859759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5367e84b-17a8-4776-8254-f6edf0de77c9-data-volume\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.859901 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.859846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5367e84b-17a8-4776-8254-f6edf0de77c9-crio-socket\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.960972 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.960934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.961080 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.960984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5367e84b-17a8-4776-8254-f6edf0de77c9-data-volume\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.961080 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.961038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5367e84b-17a8-4776-8254-f6edf0de77c9-crio-socket\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.961151 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.961130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grt8j\" (UniqueName: \"kubernetes.io/projected/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-api-access-grt8j\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.961204 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.961161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.961252 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.961188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5367e84b-17a8-4776-8254-f6edf0de77c9-crio-socket\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.961252 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:59.961225    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:50:59.961252 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:59.961248    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:50:59.961386 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.961287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5367e84b-17a8-4776-8254-f6edf0de77c9-data-volume\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.961386 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:59.961305    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.461284698 +0000 UTC m=+97.361510083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : configmap "kube-rbac-proxy" not found
Jul 13 18:50:59.961386 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:50:59.961327    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.46131585 +0000 UTC m=+97.361541221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:50:59.973023 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.973002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grt8j\" (UniqueName: \"kubernetes.io/projected/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-api-access-grt8j\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:50:59.979155 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:50:59.979079    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.083678 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:00.083643    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.191069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:00.191042    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.321426 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:00.321346    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.443236 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:00.443202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.465383 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:00.465353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:00.465517 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:00.465391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:00.465517 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:00.465480    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:00.465517 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:00.465509    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:00.465640 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:00.465540    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.465521268 +0000 UTC m=+98.365746654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:00.465640 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:00.465569    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.465554153 +0000 UTC m=+98.365779522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:00.731097 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:00.731058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:01.470656 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:01.470629    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pd2tc_2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf/dns-node-resolver/0.log"
Jul 13 18:51:01.473917 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:01.473896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:01.474136 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:01.473930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:01.474136 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:01.474032    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:01.474136 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:01.474036    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:01.474136 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:01.474086    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.474074921 +0000 UTC m=+100.374300292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:01.474136 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:01.474098    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.474092908 +0000 UTC m=+100.374318277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:02.066162 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:02.066118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.151956 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:02.151919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.649092 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:02.649055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.777864 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:02.777830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.885192 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:02.885152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") pod \"image-registry-6dbc69b4d4-5mfjc\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") " pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:51:02.885358 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:02.885294    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:51:02.885358 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:02.885309    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dbc69b4d4-5mfjc: secret "image-registry-tls" not found
Jul 13 18:51:02.885429 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:02.885365    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls podName:2cb09706-4278-4a46-953f-d86441a40fe7 nodeName:}" failed. No retries permitted until 2026-07-13 18:52:06.885349826 +0000 UTC m=+163.785575199 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls") pod "image-registry-6dbc69b4d4-5mfjc" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7") : secret "image-registry-tls" not found
Jul 13 18:51:02.888178 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:02.888153    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ql9cx_9e61056f-7549-4981-9e2f-885a7705b066/node-ca/0.log"
Jul 13 18:51:02.985659 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:02.985627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:51:02.985821 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:02.985767    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:51:02.985865 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:02.985841    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls podName:b19da82d-b0dc-40ac-ad54-b8acae858b1a nodeName:}" failed. No retries permitted until 2026-07-13 18:52:06.985826157 +0000 UTC m=+163.886051528 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls") pod "dns-default-7f688" (UID: "b19da82d-b0dc-40ac-ad54-b8acae858b1a") : secret "dns-default-metrics-tls" not found
Jul 13 18:51:03.490944 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:03.490908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:03.490944 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:03.490945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:03.491367 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:03.491059    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:03.491367 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:03.491063    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:03.491367 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:03.491120    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:07.491105027 +0000 UTC m=+104.391330412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:03.491367 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:03.491135    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:07.49112933 +0000 UTC m=+104.391354700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:03.668518 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:03.668489    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wnkpx_1bea1125-1479-43d3-a72b-411ddb57cd13/serve-healthcheck-canary/0.log"
Jul 13 18:51:03.829995 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:03.829915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:04.542770 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:04.542739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:04.701719 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:04.701686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:04.802084 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:04.801988    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:04.911358 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:04.911326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5l5l6"
Jul 13 18:51:04.912548 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:04.912526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:05.081879 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:05.081780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:05.192235 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:05.192205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:05.302474 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:05.302440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:06.028977 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:06.028940    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:07.522808 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:07.522749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:07.523214 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:07.522874    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:07.523214 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:07.522930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:07.523214 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:07.522942    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.522922813 +0000 UTC m=+112.423148187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:07.523214 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:07.523033    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:07.523214 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:07.523082    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls podName:5367e84b-17a8-4776-8254-f6edf0de77c9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.523067712 +0000 UTC m=+112.423293087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmgq" (UID: "5367e84b-17a8-4776-8254-f6edf0de77c9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:07.686068 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:07.686033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:07.772341 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:07.772311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:07.860926 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:07.860843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.030697 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:08.030659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.251715 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:08.251679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.428596 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:08.428567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.575940 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:08.575856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.682315 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:08.682274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.850691 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:08.850607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.235467 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:09.235432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.320280 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:09.320249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.836673 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:09.836645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.064491 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:10.064462    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.453442 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:10.453413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.650379 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:10.650344    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.845721 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:10.845691    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.957293 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:10.957258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:11.831983 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:11.831948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.224131 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:12.224093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.428882 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:12.428849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.631352 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:12.631258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.820641 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:12.820609    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.944962 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:12.944866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:13.828719 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:13.828689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:14.022460 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:14.022427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:14.430222 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:14.430194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:14.618646 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:14.618605    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:15.027260 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.027209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:15.242976 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.242943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:15.585216 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.585182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:15.585216 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.585216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:15.585723 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.585705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5367e84b-17a8-4776-8254-f6edf0de77c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:15.587578 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.587560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5367e84b-17a8-4776-8254-f6edf0de77c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmgq\" (UID: \"5367e84b-17a8-4776-8254-f6edf0de77c9\") " pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:15.615545 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.615518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kmgq"
Jul 13 18:51:15.742599 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.742569    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kmgq"]
Jul 13 18:51:15.745651 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:51:15.745626    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5367e84b_17a8_4776_8254_f6edf0de77c9.slice/crio-da6ebc013c6e505cc33abc2f7c7442d97a4760ca13e976bd1d0eabb7207b8c65 WatchSource:0}: Error finding container da6ebc013c6e505cc33abc2f7c7442d97a4760ca13e976bd1d0eabb7207b8c65: Status 404 returned error can't find the container with id da6ebc013c6e505cc33abc2f7c7442d97a4760ca13e976bd1d0eabb7207b8c65
Jul 13 18:51:15.812362 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:15.812317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:16.028131 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:16.028095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmgq" event={"ID":"5367e84b-17a8-4776-8254-f6edf0de77c9","Type":"ContainerStarted","Data":"da6ebc013c6e505cc33abc2f7c7442d97a4760ca13e976bd1d0eabb7207b8c65"}
Jul 13 18:51:20.041876 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:20.041845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmgq" event={"ID":"5367e84b-17a8-4776-8254-f6edf0de77c9","Type":"ContainerStarted","Data":"2ba87fdb0313560a31100e62566a854c76663225310221b9c92b13b64d38f3fb"}
Jul 13 18:51:20.418057 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:20.417962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:20.517605 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:20.517571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:20.606570 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:20.606522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:20.705982 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:20.705948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:20.896495 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:20.896458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:21.076390 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:21.076361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:21.180851 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:21.180817    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:22.054768 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:22.054722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmgq" event={"ID":"5367e84b-17a8-4776-8254-f6edf0de77c9","Type":"ContainerStarted","Data":"a17d608b6d0ab28f14f0a27a403040c5db46e6eefbe2befaa02411f00e22d4b0"}
Jul 13 18:51:22.094811 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:22.094757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:22.298562 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:22.298526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:22.592776 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:22.592740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:24.204485 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:24.204451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:24.304443 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:24.304410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:25.066371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:25.066281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmgq" event={"ID":"5367e84b-17a8-4776-8254-f6edf0de77c9","Type":"ContainerStarted","Data":"6e1ef0df7df07a508c64ade0e2496dbc9c39cd57c984f984859c8f4c11ff4d3d"}
Jul 13 18:51:25.096941 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:25.096897    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7kmgq" podStartSLOduration=17.153020525 podStartE2EDuration="26.096880836s" totalImagesPullingTime="8.943860311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:25.096335992 +0000 UTC m=+121.996561384" watchObservedRunningTime="2026-07-13 18:51:25.096880836 +0000 UTC m=+121.997106228"
Jul 13 18:51:28.095221 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:28.095181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:28.190243 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:28.190206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:28.401441 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:28.401352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:28.522242 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:28.522210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:28.731285 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:28.731253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:28.830600 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:28.830569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:29.278726 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:29.278686    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" podUID="d3d722ad-8eec-4a87-abe2-38136057f087" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 13 18:51:30.067750 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:30.067684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:30.262560 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:30.262532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:30.419652 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:30.419567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:30.915867 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:30.915833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:31.134202 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:31.134166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:31.197485 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:31.197396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:33.421070 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:33.421027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:51:33.423415 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:33.423394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e985b1a-0340-49e6-be50-39cf1ae8fe7c-metrics-certs\") pod \"network-metrics-daemon-45fpx\" (UID: \"1e985b1a-0340-49e6-be50-39cf1ae8fe7c\") " pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:51:33.523027 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:33.523001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gsq9l\""
Jul 13 18:51:33.531581 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:33.531563    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45fpx"
Jul 13 18:51:33.662990 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:33.662962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-45fpx"]
Jul 13 18:51:33.665734 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:51:33.665706    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e985b1a_0340_49e6_be50_39cf1ae8fe7c.slice/crio-057ce4f7015a89a387dbfe52f415cb01ee083fb08a60d9fec36eb8906f38cd38 WatchSource:0}: Error finding container 057ce4f7015a89a387dbfe52f415cb01ee083fb08a60d9fec36eb8906f38cd38: Status 404 returned error can't find the container with id 057ce4f7015a89a387dbfe52f415cb01ee083fb08a60d9fec36eb8906f38cd38
Jul 13 18:51:34.034563 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:34.034524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:34.094406 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:34.094372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-45fpx" event={"ID":"1e985b1a-0340-49e6-be50-39cf1ae8fe7c","Type":"ContainerStarted","Data":"057ce4f7015a89a387dbfe52f415cb01ee083fb08a60d9fec36eb8906f38cd38"}
Jul 13 18:51:35.098705 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:35.098625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-45fpx" event={"ID":"1e985b1a-0340-49e6-be50-39cf1ae8fe7c","Type":"ContainerStarted","Data":"293d271f681d45476d0483fdb0e28f5f650c825f42179cc435f84ff60dee99fd"}
Jul 13 18:51:35.098705 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:35.098688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-45fpx" event={"ID":"1e985b1a-0340-49e6-be50-39cf1ae8fe7c","Type":"ContainerStarted","Data":"b1e396b8edc0aa50c613492a672f9d47106e2d4ada3cf47058370d68b0fe874c"}
Jul 13 18:51:35.118581 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:35.118531    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-45fpx" podStartSLOduration=129.988010406 podStartE2EDuration="2m11.118515433s" totalImagesPullingTime="1.130505027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:35.117442164 +0000 UTC m=+132.017667557" watchObservedRunningTime="2026-07-13 18:51:35.118515433 +0000 UTC m=+132.018740889"
Jul 13 18:51:35.227003 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:35.226970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:35.306456 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:35.306425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:35.509283 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:35.509254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:35.606688 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:35.606654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:36.277890 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:36.277854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:36.407094 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:36.407054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:37.398854 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:37.398820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:37.485941 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:37.485908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:37.605029 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:37.604995    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:39.279387 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:39.279343    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" podUID="d3d722ad-8eec-4a87-abe2-38136057f087" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 13 18:51:39.741405 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:39.741374    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:39.842322 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:39.842288    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:42.635766 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:42.635732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:42.896772 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:42.896676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:42.993703 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:42.993674    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hb676"]
Jul 13 18:51:42.997307 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:42.997288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:42.999675 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:42.999650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zmqhc\""
Jul 13 18:51:43.000228 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.000209    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 18:51:43.000325 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.000213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 18:51:43.000325 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.000211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 18:51:43.000573 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.000561    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 18:51:43.000746 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.000732    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 18:51:43.000812 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.000732    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 18:51:43.033303 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.033278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.103866 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.103835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e38820-4feb-490b-b66b-d72560a2ca58-metrics-client-ca\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.103866 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.103870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.104049 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.103902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-root\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.104049 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.103921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-textfile\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.104049 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.103966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-tls\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.104049 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.104013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-sys\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.104049 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.104031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-wtmp\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.104201 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.104068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-accelerators-collector-config\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.104201 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.104094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7mf\" (UniqueName: \"kubernetes.io/projected/25e38820-4feb-490b-b66b-d72560a2ca58-kube-api-access-ml7mf\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.131279 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.131245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.204816 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.204714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-root\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.204816 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.204750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-textfile\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.204816 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.204767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-tls\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.204851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-root\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.204893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-sys\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:43.204899    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.204912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-wtmp\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:43.204960    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-tls podName:25e38820-4feb-490b-b66b-d72560a2ca58 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.704939183 +0000 UTC m=+140.605164554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-tls") pod "node-exporter-hb676" (UID: "25e38820-4feb-490b-b66b-d72560a2ca58") : secret "node-exporter-tls" not found
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.204998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-accelerators-collector-config\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-sys\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-wtmp\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7mf\" (UniqueName: \"kubernetes.io/projected/25e38820-4feb-490b-b66b-d72560a2ca58-kube-api-access-ml7mf\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205373 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e38820-4feb-490b-b66b-d72560a2ca58-metrics-client-ca\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205373 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-textfile\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205373 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205551 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-accelerators-collector-config\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.205666 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.205557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25e38820-4feb-490b-b66b-d72560a2ca58-metrics-client-ca\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.207424 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.207405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.220221 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.220194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7mf\" (UniqueName: \"kubernetes.io/projected/25e38820-4feb-490b-b66b-d72560a2ca58-kube-api-access-ml7mf\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.247733 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.247703    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.344305 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.344268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.446299 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.446261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.560675 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.560635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.710990 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.710957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-tls\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.713279 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.713260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25e38820-4feb-490b-b66b-d72560a2ca58-node-exporter-tls\") pod \"node-exporter-hb676\" (UID: \"25e38820-4feb-490b-b66b-d72560a2ca58\") " pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.805115 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.805082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:43.905820 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.905737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hb676"
Jul 13 18:51:43.915948 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:51:43.915922    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e38820_4feb_490b_b66b_d72560a2ca58.slice/crio-6aa7ae080bcd7eb65075654a356c25d1fe122f45f423d553cb4ffe6ca8e5d466 WatchSource:0}: Error finding container 6aa7ae080bcd7eb65075654a356c25d1fe122f45f423d553cb4ffe6ca8e5d466: Status 404 returned error can't find the container with id 6aa7ae080bcd7eb65075654a356c25d1fe122f45f423d553cb4ffe6ca8e5d466
Jul 13 18:51:43.920561 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:43.920513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:44.127079 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:44.127049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hb676" event={"ID":"25e38820-4feb-490b-b66b-d72560a2ca58","Type":"ContainerStarted","Data":"6aa7ae080bcd7eb65075654a356c25d1fe122f45f423d553cb4ffe6ca8e5d466"}
Jul 13 18:51:44.146879 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:44.146843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:45.416166 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:45.416126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:45.509340 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:45.509292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:45.889663 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:45.889630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:46.005047 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:46.005014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:46.119629 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:46.119595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:46.214270 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:46.214198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:46.495508 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:46.495418    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:47.137713 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:47.137683    2581 generic.go:356] "Generic (PLEG): container finished" podID="25e38820-4feb-490b-b66b-d72560a2ca58" containerID="2ddbb4b5203e82e575f8b64881d1730c489a7487bd4a8fff88075854234d2829" exitCode=0
Jul 13 18:51:47.137885 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:47.137769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hb676" event={"ID":"25e38820-4feb-490b-b66b-d72560a2ca58","Type":"ContainerDied","Data":"2ddbb4b5203e82e575f8b64881d1730c489a7487bd4a8fff88075854234d2829"}
Jul 13 18:51:47.620828 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:47.620776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:47.718295 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:47.718265    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:47.819025 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:47.818952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:48.142940 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:48.142908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hb676" event={"ID":"25e38820-4feb-490b-b66b-d72560a2ca58","Type":"ContainerStarted","Data":"bff3443b6664f486a2bcee1e78645fe2f1a28be963a3e8bcb937c2c6ad2358eb"}
Jul 13 18:51:48.143103 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:48.142958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hb676" event={"ID":"25e38820-4feb-490b-b66b-d72560a2ca58","Type":"ContainerStarted","Data":"e88ea579e125026372c3a0a0c2f78d93348c1988e03f94e14aa571fb4f8c983a"}
Jul 13 18:51:48.167752 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:48.167290    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hb676" podStartSLOduration=3.700173755 podStartE2EDuration="6.167270179s" totalImagesPullingTime="2.396598142s" totalInitContainerRuntime="70.498282ms" isStatefulPod=true podCreationTimestamp="2026-07-13 18:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:48.16622688 +0000 UTC m=+145.066452271" watchObservedRunningTime="2026-07-13 18:51:48.167270179 +0000 UTC m=+145.067495570"
Jul 13 18:51:49.250960 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:49.250913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:49.278750 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:49.278716    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" podUID="d3d722ad-8eec-4a87-abe2-38136057f087" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 13 18:51:49.278931 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:49.278782    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h"
Jul 13 18:51:49.279272 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:49.279244    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"3e3db4d12eb560093312bec6deaa64f2ce4674a74b025ac45654cca0c3849fe7"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 13 18:51:49.279312 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:49.279294    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" podUID="d3d722ad-8eec-4a87-abe2-38136057f087" containerName="service-proxy" containerID="cri-o://3e3db4d12eb560093312bec6deaa64f2ce4674a74b025ac45654cca0c3849fe7" gracePeriod=30
Jul 13 18:51:50.150645 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.150611    2581 generic.go:356] "Generic (PLEG): container finished" podID="d3d722ad-8eec-4a87-abe2-38136057f087" containerID="3e3db4d12eb560093312bec6deaa64f2ce4674a74b025ac45654cca0c3849fe7" exitCode=2
Jul 13 18:51:50.150822 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.150652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" event={"ID":"d3d722ad-8eec-4a87-abe2-38136057f087","Type":"ContainerDied","Data":"3e3db4d12eb560093312bec6deaa64f2ce4674a74b025ac45654cca0c3849fe7"}
Jul 13 18:51:50.150822 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.150684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b78dcd44c-vpc9h" event={"ID":"d3d722ad-8eec-4a87-abe2-38136057f087","Type":"ContainerStarted","Data":"1fd88ac769d398479f8cf5bc83dd22dd79d505bb4cdb5b5706848753e2dc41d6"}
Jul 13 18:51:50.433866 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.433769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:50.516248 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.516214    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"]
Jul 13 18:51:50.516455 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:51:50.516436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc" podUID="2cb09706-4278-4a46-953f-d86441a40fe7"
Jul 13 18:51:50.529841 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.529811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:50.622129 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.622103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:50.712492 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:50.712461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:51.153308 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.153230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:51:51.159480 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.159454    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:51:51.270647 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.270614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-image-registry-private-configuration\") pod \"2cb09706-4278-4a46-953f-d86441a40fe7\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") "
Jul 13 18:51:51.270845 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.270667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-trusted-ca\") pod \"2cb09706-4278-4a46-953f-d86441a40fe7\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") "
Jul 13 18:51:51.270845 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.270701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-bound-sa-token\") pod \"2cb09706-4278-4a46-953f-d86441a40fe7\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") "
Jul 13 18:51:51.270845 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.270736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cb09706-4278-4a46-953f-d86441a40fe7-ca-trust-extracted\") pod \"2cb09706-4278-4a46-953f-d86441a40fe7\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") "
Jul 13 18:51:51.270845 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.270770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-installation-pull-secrets\") pod \"2cb09706-4278-4a46-953f-d86441a40fe7\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") "
Jul 13 18:51:51.271061 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.270845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-registry-certificates\") pod \"2cb09706-4278-4a46-953f-d86441a40fe7\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") "
Jul 13 18:51:51.271061 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.270880    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfr5z\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-kube-api-access-cfr5z\") pod \"2cb09706-4278-4a46-953f-d86441a40fe7\" (UID: \"2cb09706-4278-4a46-953f-d86441a40fe7\") "
Jul 13 18:51:51.271164 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.271081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb09706-4278-4a46-953f-d86441a40fe7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2cb09706-4278-4a46-953f-d86441a40fe7" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:51:51.271264 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.271241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2cb09706-4278-4a46-953f-d86441a40fe7" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:51.271354 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.271255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2cb09706-4278-4a46-953f-d86441a40fe7" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:51.273219 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.273189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2cb09706-4278-4a46-953f-d86441a40fe7" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:51.273322 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.273278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-kube-api-access-cfr5z" (OuterVolumeSpecName: "kube-api-access-cfr5z") pod "2cb09706-4278-4a46-953f-d86441a40fe7" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7"). InnerVolumeSpecName "kube-api-access-cfr5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:51:51.273322 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.273291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2cb09706-4278-4a46-953f-d86441a40fe7" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:51.273396 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.273331    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2cb09706-4278-4a46-953f-d86441a40fe7" (UID: "2cb09706-4278-4a46-953f-d86441a40fe7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:51:51.371751 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.371719    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-registry-certificates\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:51.371751 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.371744    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfr5z\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-kube-api-access-cfr5z\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:51.371751 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.371753    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-image-registry-private-configuration\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:51.371974 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.371761    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cb09706-4278-4a46-953f-d86441a40fe7-trusted-ca\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:51.371974 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.371771    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-bound-sa-token\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:51.371974 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.371780    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cb09706-4278-4a46-953f-d86441a40fe7-ca-trust-extracted\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:51.371974 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:51.371800    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cb09706-4278-4a46-953f-d86441a40fe7-installation-pull-secrets\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:52.149584 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:52.149550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:52.155702 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:52.155674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"
Jul 13 18:51:52.196680 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:52.196647    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"]
Jul 13 18:51:52.202004 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:52.201982    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dbc69b4d4-5mfjc"]
Jul 13 18:51:52.250805 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:52.250754    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:52.378495 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:52.378465    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cb09706-4278-4a46-953f-d86441a40fe7-registry-tls\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.243536 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:53.243504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:53.314180 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:53.314138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:53.602159 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:53.602085    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cb09706-4278-4a46-953f-d86441a40fe7" path="/var/lib/kubelet/pods/2cb09706-4278-4a46-953f-d86441a40fe7/volumes"
Jul 13 18:51:53.871696 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:53.871609    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:53.976814 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:53.976763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:54.073161 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:51:54.073132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.038398 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:52:02.038349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7f688" podUID="b19da82d-b0dc-40ac-ad54-b8acae858b1a"
Jul 13 18:52:02.185954 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:02.185923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7f688"
Jul 13 18:52:02.800174 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:02.800133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.901167 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:02.901120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:02.966999 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:02.966970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:07.085449 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:07.085414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:52:07.087914 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:07.087895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19da82d-b0dc-40ac-ad54-b8acae858b1a-metrics-tls\") pod \"dns-default-7f688\" (UID: \"b19da82d-b0dc-40ac-ad54-b8acae858b1a\") " pod="openshift-dns/dns-default-7f688"
Jul 13 18:52:07.289012 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:07.288981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-88cff\""
Jul 13 18:52:07.297548 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:07.297528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7f688"
Jul 13 18:52:07.445538 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:07.445503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7f688"]
Jul 13 18:52:07.451978 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:52:07.451943    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19da82d_b0dc_40ac_ad54_b8acae858b1a.slice/crio-76d5d9d020d62275ed50fcd879b152e140d77cc388a71085dd5b75ca2d17f544 WatchSource:0}: Error finding container 76d5d9d020d62275ed50fcd879b152e140d77cc388a71085dd5b75ca2d17f544: Status 404 returned error can't find the container with id 76d5d9d020d62275ed50fcd879b152e140d77cc388a71085dd5b75ca2d17f544
Jul 13 18:52:08.205958 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:08.205916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7f688" event={"ID":"b19da82d-b0dc-40ac-ad54-b8acae858b1a","Type":"ContainerStarted","Data":"76d5d9d020d62275ed50fcd879b152e140d77cc388a71085dd5b75ca2d17f544"}
Jul 13 18:52:08.239031 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:08.238994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:08.308763 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:08.308709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:09.210623 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:09.210591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7f688" event={"ID":"b19da82d-b0dc-40ac-ad54-b8acae858b1a","Type":"ContainerStarted","Data":"cb1873f7a7e95f1db6768ca634568638e48378d51cd25b21f01eeb6b31564226"}
Jul 13 18:52:09.211024 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:09.210642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7f688" event={"ID":"b19da82d-b0dc-40ac-ad54-b8acae858b1a","Type":"ContainerStarted","Data":"07cc5e35df1d364655e40e88810c9ecb6ad8dc01b66a52c4d4eb30443fc4e0d1"}
Jul 13 18:52:09.211024 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:09.210704    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7f688"
Jul 13 18:52:09.229187 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:09.229140    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7f688" podStartSLOduration=129.732821837 podStartE2EDuration="2m11.229124788s" totalImagesPullingTime="1.496302951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:09.227932278 +0000 UTC m=+166.128157662" watchObservedRunningTime="2026-07-13 18:52:09.229124788 +0000 UTC m=+166.129350179"
Jul 13 18:52:19.216474 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:19.216394    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7f688"
Jul 13 18:52:21.043634 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:21.043605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wnkpx_1bea1125-1479-43d3-a72b-411ddb57cd13/serve-healthcheck-canary/0.log"
Jul 13 18:52:28.368475 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:28.368441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:28.469402 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:28.469359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:33.934277 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:33.934247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:34.066803 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:34.066754    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:34.166168 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:34.166129    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:49.382435 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:52:49.382399    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:02.233243 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:02.233202    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:02.433947 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:02.433918    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:10.741151 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:10.741114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:10.813094 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:10.813056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:22.176362 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:22.176323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:22.277216 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:22.277183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:22.343382 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:22.343179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:35.261017 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:35.260982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:51.034563 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:51.034468    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:51.092387 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:53:51.092354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:17.513127 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:17.513088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:17.572389 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:17.572350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:20.685626 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:20.685597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:23.515701 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:23.515670    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:54:23.519769 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:23.519744    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:54:23.521062 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:23.521047    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 18:54:29.561383 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:29.561340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:29.619631 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:29.619597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:31.447573 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:31.447528    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:31.548326 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:31.548290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:31.621944 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:31.621913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:38.303220 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:38.303181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:38.403291 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:38.403253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:59.848282 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:54:59.848245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:04.632208 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:04.632170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:06.557381 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:06.557342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:08.371436 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:08.371395    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:08.633546 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:08.633460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:08.787216 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:08.787171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:08.892598 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:08.892505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:08.951706 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:08.951673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:09.831010 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.830979    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"]
Jul 13 18:56:09.833932 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.833916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:09.835940 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.835919    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6cwfd\""
Jul 13 18:56:09.837873 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.837855    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 13 18:56:09.837988 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.837896    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 13 18:56:09.846243 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.846217    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"]
Jul 13 18:56:09.931949 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.931911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ld9b\" (UniqueName: \"kubernetes.io/projected/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-kube-api-access-7ld9b\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:09.932126 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.931964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:09.932126 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:09.932058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.033076 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.033035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ld9b\" (UniqueName: \"kubernetes.io/projected/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-kube-api-access-7ld9b\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.033076 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.033081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.033272 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.033108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.033440 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.033425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.033484 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.033461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.045235 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.045199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ld9b\" (UniqueName: \"kubernetes.io/projected/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-kube-api-access-7ld9b\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.142233 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.142145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:10.267919 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.267885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"]
Jul 13 18:56:10.270859 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:56:10.270828    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e0dbd7_ffaf_4f36_ad8d_920abdfc4072.slice/crio-9d7cf677288d96665ba735612429ab59cca3be39a0143015001570680422feb9 WatchSource:0}: Error finding container 9d7cf677288d96665ba735612429ab59cca3be39a0143015001570680422feb9: Status 404 returned error can't find the container with id 9d7cf677288d96665ba735612429ab59cca3be39a0143015001570680422feb9
Jul 13 18:56:10.272819 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.272778    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:56:10.977946 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:10.977900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc" event={"ID":"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072","Type":"ContainerStarted","Data":"9d7cf677288d96665ba735612429ab59cca3be39a0143015001570680422feb9"}
Jul 13 18:56:17.001151 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:17.001115    2581 generic.go:356] "Generic (PLEG): container finished" podID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerID="01e6c7eb073c4ba742f9e6bdcb55c9e536b03e1fd5d45553f029c0a3e39d6f24" exitCode=0
Jul 13 18:56:17.001706 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:17.001163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc" event={"ID":"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072","Type":"ContainerDied","Data":"01e6c7eb073c4ba742f9e6bdcb55c9e536b03e1fd5d45553f029c0a3e39d6f24"}
Jul 13 18:56:20.012371 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:20.012336    2581 generic.go:356] "Generic (PLEG): container finished" podID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerID="61a1cb2711d4428f8082c226d8bcc5c691f8b3fc054a633be25487b20372de96" exitCode=0
Jul 13 18:56:20.012735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:20.012387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc" event={"ID":"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072","Type":"ContainerDied","Data":"61a1cb2711d4428f8082c226d8bcc5c691f8b3fc054a633be25487b20372de96"}
Jul 13 18:56:28.043649 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:28.043610    2581 generic.go:356] "Generic (PLEG): container finished" podID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerID="a7d220cd6233209b15bd2f23e84f4f4142bc519d197d4447c6d4dedeae57ea25" exitCode=0
Jul 13 18:56:28.044090 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:28.043697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc" event={"ID":"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072","Type":"ContainerDied","Data":"a7d220cd6233209b15bd2f23e84f4f4142bc519d197d4447c6d4dedeae57ea25"}
Jul 13 18:56:29.169661 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.169639    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:29.266947 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.266918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-bundle\") pod \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") "
Jul 13 18:56:29.267140 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.266960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ld9b\" (UniqueName: \"kubernetes.io/projected/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-kube-api-access-7ld9b\") pod \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") "
Jul 13 18:56:29.267140 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.266984    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-util\") pod \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\" (UID: \"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072\") "
Jul 13 18:56:29.267621 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.267591    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-bundle" (OuterVolumeSpecName: "bundle") pod "76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" (UID: "76e0dbd7-ffaf-4f36-ad8d-920abdfc4072"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:56:29.269300 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.269274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-kube-api-access-7ld9b" (OuterVolumeSpecName: "kube-api-access-7ld9b") pod "76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" (UID: "76e0dbd7-ffaf-4f36-ad8d-920abdfc4072"). InnerVolumeSpecName "kube-api-access-7ld9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:56:29.271098 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.271080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-util" (OuterVolumeSpecName: "util") pod "76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" (UID: "76e0dbd7-ffaf-4f36-ad8d-920abdfc4072"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:56:29.368241 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.368158    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ld9b\" (UniqueName: \"kubernetes.io/projected/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-kube-api-access-7ld9b\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:56:29.368241 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.368186    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-util\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:56:29.368241 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:29.368198    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e0dbd7-ffaf-4f36-ad8d-920abdfc4072-bundle\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:56:30.052605 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:30.052579    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc"
Jul 13 18:56:30.052819 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:30.052582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5dnlbc" event={"ID":"76e0dbd7-ffaf-4f36-ad8d-920abdfc4072","Type":"ContainerDied","Data":"9d7cf677288d96665ba735612429ab59cca3be39a0143015001570680422feb9"}
Jul 13 18:56:30.052819 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:30.052699    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7cf677288d96665ba735612429ab59cca3be39a0143015001570680422feb9"
Jul 13 18:56:31.200334 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:31.200292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:31.297015 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:31.296983    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:32.820705 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820671    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"]
Jul 13 18:56:32.821093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820906    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerName="extract"
Jul 13 18:56:32.821093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerName="extract"
Jul 13 18:56:32.821093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820927    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerName="util"
Jul 13 18:56:32.821093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820933    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerName="util"
Jul 13 18:56:32.821093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820948    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerName="pull"
Jul 13 18:56:32.821093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820954    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerName="pull"
Jul 13 18:56:32.821093 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.820992    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e0dbd7-ffaf-4f36-ad8d-920abdfc4072" containerName="extract"
Jul 13 18:56:32.865548 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.865515    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"]
Jul 13 18:56:32.865711 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.865634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:32.867722 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.867700    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 13 18:56:32.867856 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.867835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 13 18:56:32.867856 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.867841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qfnvb\""
Jul 13 18:56:32.867856 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.867835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 13 18:56:32.891943 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.891915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/aec314d0-6448-4dcb-a7f2-1b5373463464-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2\" (UID: \"aec314d0-6448-4dcb-a7f2-1b5373463464\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:32.892076 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.891961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brjq\" (UniqueName: \"kubernetes.io/projected/aec314d0-6448-4dcb-a7f2-1b5373463464-kube-api-access-6brjq\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2\" (UID: \"aec314d0-6448-4dcb-a7f2-1b5373463464\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:32.992777 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.992743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/aec314d0-6448-4dcb-a7f2-1b5373463464-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2\" (UID: \"aec314d0-6448-4dcb-a7f2-1b5373463464\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:32.992974 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.992821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brjq\" (UniqueName: \"kubernetes.io/projected/aec314d0-6448-4dcb-a7f2-1b5373463464-kube-api-access-6brjq\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2\" (UID: \"aec314d0-6448-4dcb-a7f2-1b5373463464\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:32.995258 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:32.995226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/aec314d0-6448-4dcb-a7f2-1b5373463464-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2\" (UID: \"aec314d0-6448-4dcb-a7f2-1b5373463464\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:33.003262 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:33.003239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brjq\" (UniqueName: \"kubernetes.io/projected/aec314d0-6448-4dcb-a7f2-1b5373463464-kube-api-access-6brjq\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2\" (UID: \"aec314d0-6448-4dcb-a7f2-1b5373463464\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:33.174910 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:33.174815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:33.328484 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:33.328450    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"]
Jul 13 18:56:33.331915 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:56:33.331885    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec314d0_6448_4dcb_a7f2_1b5373463464.slice/crio-cc1adbb3c4a7d6565db3ffc2e394d152a78b81cde2f9872838ec5e60758a56c3 WatchSource:0}: Error finding container cc1adbb3c4a7d6565db3ffc2e394d152a78b81cde2f9872838ec5e60758a56c3: Status 404 returned error can't find the container with id cc1adbb3c4a7d6565db3ffc2e394d152a78b81cde2f9872838ec5e60758a56c3
Jul 13 18:56:34.067229 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:34.067192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2" event={"ID":"aec314d0-6448-4dcb-a7f2-1b5373463464","Type":"ContainerStarted","Data":"cc1adbb3c4a7d6565db3ffc2e394d152a78b81cde2f9872838ec5e60758a56c3"}
Jul 13 18:56:36.905024 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:36.904987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:37.079860 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.079825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2" event={"ID":"aec314d0-6448-4dcb-a7f2-1b5373463464","Type":"ContainerStarted","Data":"0fc10b58a16ddd2dca61b5554656cfe38aea258a2eedd1a989efd67d7cf8bd11"}
Jul 13 18:56:37.080021 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.079956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:56:37.107954 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.107891    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2" podStartSLOduration=2.000076185 podStartE2EDuration="5.10787366s" totalImagesPullingTime="3.107797475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:56:37.106867793 +0000 UTC m=+434.007093186" watchObservedRunningTime="2026-07-13 18:56:37.10787366 +0000 UTC m=+434.008099069"
Jul 13 18:56:37.245737 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.245702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:37.471069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.471038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:37.573535 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.573448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:37.660298 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.660264    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"]
Jul 13 18:56:37.663357 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.663342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.665553 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.665522    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-p6zqz\""
Jul 13 18:56:37.665689 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.665525    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 13 18:56:37.665819 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.665728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 13 18:56:37.666141 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.666106    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:37.681874 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.681844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"]
Jul 13 18:56:37.731006 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.730971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.731006 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.731007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmvx\" (UniqueName: \"kubernetes.io/projected/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-kube-api-access-hqmvx\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.731220 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.731074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-certificates\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.832254 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.832174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.832254 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.832213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmvx\" (UniqueName: \"kubernetes.io/projected/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-kube-api-access-hqmvx\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.832254 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.832245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-certificates\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.832630 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.832608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.834838 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.834823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-certificates\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.841088 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.841061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmvx\" (UniqueName: \"kubernetes.io/projected/dfd07bb1-1e8f-4fa7-ac31-cebab66950f3-kube-api-access-hqmvx\") pod \"keda-metrics-apiserver-55557df8c-td5fs\" (UID: \"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.973866 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.973831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:37.975229 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.975205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:37.982736 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.982714    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-fq48f"]
Jul 13 18:56:37.986013 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.985991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:37.988913 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:37.988893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 13 18:56:38.000040 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.000017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-fq48f"]
Jul 13 18:56:38.033868 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.033838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-certificates\") pod \"keda-admission-56f9798bb4-fq48f\" (UID: \"1c75dd38-8792-4b0f-86d6-9dccf8a460e4\") " pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:38.034005 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.033895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdvf\" (UniqueName: \"kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-kube-api-access-tcdvf\") pod \"keda-admission-56f9798bb4-fq48f\" (UID: \"1c75dd38-8792-4b0f-86d6-9dccf8a460e4\") " pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:38.112560 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.112491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"]
Jul 13 18:56:38.116434 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:56:38.116408    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd07bb1_1e8f_4fa7_ac31_cebab66950f3.slice/crio-1fcda40e8d2bd5401ea576203951100ef94833ce0bc0a7218743ba095b968817 WatchSource:0}: Error finding container 1fcda40e8d2bd5401ea576203951100ef94833ce0bc0a7218743ba095b968817: Status 404 returned error can't find the container with id 1fcda40e8d2bd5401ea576203951100ef94833ce0bc0a7218743ba095b968817
Jul 13 18:56:38.134503 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.134479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-certificates\") pod \"keda-admission-56f9798bb4-fq48f\" (UID: \"1c75dd38-8792-4b0f-86d6-9dccf8a460e4\") " pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:38.134641 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.134531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdvf\" (UniqueName: \"kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-kube-api-access-tcdvf\") pod \"keda-admission-56f9798bb4-fq48f\" (UID: \"1c75dd38-8792-4b0f-86d6-9dccf8a460e4\") " pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:38.134641 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:56:38.134631    2581 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 13 18:56:38.134710 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:56:38.134659    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-fq48f: secret "keda-admission-webhooks-certs" not found
Jul 13 18:56:38.134743 ip-10-0-131-203 kubenswrapper[2581]: E0713 18:56:38.134722    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-certificates podName:1c75dd38-8792-4b0f-86d6-9dccf8a460e4 nodeName:}" failed. No retries permitted until 2026-07-13 18:56:38.634704343 +0000 UTC m=+435.534929714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-certificates") pod "keda-admission-56f9798bb4-fq48f" (UID: "1c75dd38-8792-4b0f-86d6-9dccf8a460e4") : secret "keda-admission-webhooks-certs" not found
Jul 13 18:56:38.144616 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.144591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdvf\" (UniqueName: \"kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-kube-api-access-tcdvf\") pod \"keda-admission-56f9798bb4-fq48f\" (UID: \"1c75dd38-8792-4b0f-86d6-9dccf8a460e4\") " pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:38.638500 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.638453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-certificates\") pod \"keda-admission-56f9798bb4-fq48f\" (UID: \"1c75dd38-8792-4b0f-86d6-9dccf8a460e4\") " pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:38.641052 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.641029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1c75dd38-8792-4b0f-86d6-9dccf8a460e4-certificates\") pod \"keda-admission-56f9798bb4-fq48f\" (UID: \"1c75dd38-8792-4b0f-86d6-9dccf8a460e4\") " pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:38.903268 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:38.903176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:39.060633 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:39.060596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-fq48f"]
Jul 13 18:56:39.063493 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:56:39.063459    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c75dd38_8792_4b0f_86d6_9dccf8a460e4.slice/crio-07793697f91afeea7e0026f377f07d3cd0485f27b72926d10b3af990e165969f WatchSource:0}: Error finding container 07793697f91afeea7e0026f377f07d3cd0485f27b72926d10b3af990e165969f: Status 404 returned error can't find the container with id 07793697f91afeea7e0026f377f07d3cd0485f27b72926d10b3af990e165969f
Jul 13 18:56:39.089332 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:39.089291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-fq48f" event={"ID":"1c75dd38-8792-4b0f-86d6-9dccf8a460e4","Type":"ContainerStarted","Data":"07793697f91afeea7e0026f377f07d3cd0485f27b72926d10b3af990e165969f"}
Jul 13 18:56:39.090766 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:39.090728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs" event={"ID":"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3","Type":"ContainerStarted","Data":"1fcda40e8d2bd5401ea576203951100ef94833ce0bc0a7218743ba095b968817"}
Jul 13 18:56:40.046909 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:40.046873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:40.172991 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:40.172948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:41.101160 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:41.101119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-fq48f" event={"ID":"1c75dd38-8792-4b0f-86d6-9dccf8a460e4","Type":"ContainerStarted","Data":"7fb034a4d1f9f6f4084a43e6fda04660a1383b55bd807b649dd953812cfddb41"}
Jul 13 18:56:41.101320 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:41.101224    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:56:41.102738 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:41.102716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs" event={"ID":"dfd07bb1-1e8f-4fa7-ac31-cebab66950f3","Type":"ContainerStarted","Data":"12c108e3d89029f151b9d6d4201bdcea844fe6dff7830b2bfaa21cb01ba52735"}
Jul 13 18:56:41.102857 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:41.102760    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:41.118990 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:41.118945    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-fq48f" podStartSLOduration=2.261532322 podStartE2EDuration="4.118928077s" totalImagesPullingTime="1.857395755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:56:41.117681996 +0000 UTC m=+438.017907388" watchObservedRunningTime="2026-07-13 18:56:41.118928077 +0000 UTC m=+438.019153468"
Jul 13 18:56:41.135560 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:41.135503    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs" podStartSLOduration=1.330276105 podStartE2EDuration="4.135486157s" totalImagesPullingTime="2.805210052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:56:41.134382237 +0000 UTC m=+438.034607630" watchObservedRunningTime="2026-07-13 18:56:41.135486157 +0000 UTC m=+438.035711551"
Jul 13 18:56:41.322488 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:41.322452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:52.110504 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:52.110429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-td5fs"
Jul 13 18:56:58.086453 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:56:58.086420    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n9z2"
Jul 13 18:57:02.056667 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:02.056631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:02.108122 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:02.108096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-fq48f"
Jul 13 18:57:08.245122 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:08.245093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:44.709452 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:44.709410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:44.780490 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:44.780444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:44.855878 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:44.855845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:44.957374 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:44.957337    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:45.239867 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:45.239834    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:45.340993 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:45.340956    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:45.397585 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:45.397549    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:45.697210 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:57:45.697134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:15.375321 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:15.375235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:15.476847 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:15.476811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:15.577406 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:15.577366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:16.117897 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:16.117859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:16.217974 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:16.217943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:22.363047 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.363011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:22.469042 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.469014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:22.629378 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.629307    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"]
Jul 13 18:58:22.632558 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.632538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.634621 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.634598    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 13 18:58:22.635132 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.635116    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 13 18:58:22.635234 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.635196    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6cwfd\""
Jul 13 18:58:22.641525 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.641503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"]
Jul 13 18:58:22.687335 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.687298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxf5\" (UniqueName: \"kubernetes.io/projected/706f4a9c-fc47-4706-b956-014598801788-kube-api-access-sqxf5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.687507 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.687359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.687507 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.687377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.788180 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.788137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxf5\" (UniqueName: \"kubernetes.io/projected/706f4a9c-fc47-4706-b956-014598801788-kube-api-access-sqxf5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.788370 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.788212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.788370 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.788236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.788640 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.788620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.788713 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.788653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.796845 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.796814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxf5\" (UniqueName: \"kubernetes.io/projected/706f4a9c-fc47-4706-b956-014598801788-kube-api-access-sqxf5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:22.941835 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:22.941712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:23.070505 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:23.070468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"]
Jul 13 18:58:23.073233 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:58:23.073206    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706f4a9c_fc47_4706_b956_014598801788.slice/crio-df2357c3b09717580ce1e68d72ed66610bcca8578eefc123abf20ccb40d00dc0 WatchSource:0}: Error finding container df2357c3b09717580ce1e68d72ed66610bcca8578eefc123abf20ccb40d00dc0: Status 404 returned error can't find the container with id df2357c3b09717580ce1e68d72ed66610bcca8578eefc123abf20ccb40d00dc0
Jul 13 18:58:23.459907 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:23.459875    2581 generic.go:356] "Generic (PLEG): container finished" podID="706f4a9c-fc47-4706-b956-014598801788" containerID="03afb60c2b81b50d9b5731f78ec4b17a4c87560bb4738fb4465304859a6fc517" exitCode=0
Jul 13 18:58:23.460283 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:23.459961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55" event={"ID":"706f4a9c-fc47-4706-b956-014598801788","Type":"ContainerDied","Data":"03afb60c2b81b50d9b5731f78ec4b17a4c87560bb4738fb4465304859a6fc517"}
Jul 13 18:58:23.460283 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:23.460017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55" event={"ID":"706f4a9c-fc47-4706-b956-014598801788","Type":"ContainerStarted","Data":"df2357c3b09717580ce1e68d72ed66610bcca8578eefc123abf20ccb40d00dc0"}
Jul 13 18:58:25.468622 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:25.468592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55" event={"ID":"706f4a9c-fc47-4706-b956-014598801788","Type":"ContainerStarted","Data":"cee2d17e5fdb0e370194d3498a0f2450745c0dc1f20c2377de86d2e8e7dfb19d"}
Jul 13 18:58:26.444805 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:26.444753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:26.474303 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:26.474268    2581 generic.go:356] "Generic (PLEG): container finished" podID="706f4a9c-fc47-4706-b956-014598801788" containerID="cee2d17e5fdb0e370194d3498a0f2450745c0dc1f20c2377de86d2e8e7dfb19d" exitCode=0
Jul 13 18:58:26.474657 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:26.474343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55" event={"ID":"706f4a9c-fc47-4706-b956-014598801788","Type":"ContainerDied","Data":"cee2d17e5fdb0e370194d3498a0f2450745c0dc1f20c2377de86d2e8e7dfb19d"}
Jul 13 18:58:26.501338 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:26.501310    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:26.595129 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:26.595097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:27.479684 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:27.479649    2581 generic.go:356] "Generic (PLEG): container finished" podID="706f4a9c-fc47-4706-b956-014598801788" containerID="8c51c5172f93aee2d864927df4c98540b8f7ff8a34ea7965a49814918657d2da" exitCode=0
Jul 13 18:58:27.480081 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:27.479691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55" event={"ID":"706f4a9c-fc47-4706-b956-014598801788","Type":"ContainerDied","Data":"8c51c5172f93aee2d864927df4c98540b8f7ff8a34ea7965a49814918657d2da"}
Jul 13 18:58:28.612155 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.612130    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:28.633203 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.633172    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-bundle\") pod \"706f4a9c-fc47-4706-b956-014598801788\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") "
Jul 13 18:58:28.633350 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.633242    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqxf5\" (UniqueName: \"kubernetes.io/projected/706f4a9c-fc47-4706-b956-014598801788-kube-api-access-sqxf5\") pod \"706f4a9c-fc47-4706-b956-014598801788\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") "
Jul 13 18:58:28.633350 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.633282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-util\") pod \"706f4a9c-fc47-4706-b956-014598801788\" (UID: \"706f4a9c-fc47-4706-b956-014598801788\") "
Jul 13 18:58:28.633900 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.633873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-bundle" (OuterVolumeSpecName: "bundle") pod "706f4a9c-fc47-4706-b956-014598801788" (UID: "706f4a9c-fc47-4706-b956-014598801788"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:58:28.635474 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.635446    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706f4a9c-fc47-4706-b956-014598801788-kube-api-access-sqxf5" (OuterVolumeSpecName: "kube-api-access-sqxf5") pod "706f4a9c-fc47-4706-b956-014598801788" (UID: "706f4a9c-fc47-4706-b956-014598801788"). InnerVolumeSpecName "kube-api-access-sqxf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:58:28.686756 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.686717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-util" (OuterVolumeSpecName: "util") pod "706f4a9c-fc47-4706-b956-014598801788" (UID: "706f4a9c-fc47-4706-b956-014598801788"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:58:28.733772 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.733686    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-util\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:58:28.733772 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.733719    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/706f4a9c-fc47-4706-b956-014598801788-bundle\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:58:28.733772 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:28.733735    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxf5\" (UniqueName: \"kubernetes.io/projected/706f4a9c-fc47-4706-b956-014598801788-kube-api-access-sqxf5\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:58:29.489968 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:29.489929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55" event={"ID":"706f4a9c-fc47-4706-b956-014598801788","Type":"ContainerDied","Data":"df2357c3b09717580ce1e68d72ed66610bcca8578eefc123abf20ccb40d00dc0"}
Jul 13 18:58:29.489968 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:29.489968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vxv55"
Jul 13 18:58:29.490188 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:29.490020    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df2357c3b09717580ce1e68d72ed66610bcca8578eefc123abf20ccb40d00dc0"
Jul 13 18:58:33.523985 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:33.523956    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:34.392410 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:34.392376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:37.334591 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:37.334518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:37.439778 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:37.439739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:39.667931 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:39.667874    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:39.764103 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:39.764067    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:41.661607 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:41.661568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:42.059537 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:42.059498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:42.334886 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:42.334775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:42.436858 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:42.436823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:43.939366 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:43.939328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:44.041542 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.041507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:44.141167 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.141135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:44.957975 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.957939    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"]
Jul 13 18:58:44.958343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.958183    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706f4a9c-fc47-4706-b956-014598801788" containerName="pull"
Jul 13 18:58:44.958343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.958193    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="706f4a9c-fc47-4706-b956-014598801788" containerName="pull"
Jul 13 18:58:44.958343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.958201    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706f4a9c-fc47-4706-b956-014598801788" containerName="util"
Jul 13 18:58:44.958343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.958206    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="706f4a9c-fc47-4706-b956-014598801788" containerName="util"
Jul 13 18:58:44.958343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.958212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706f4a9c-fc47-4706-b956-014598801788" containerName="extract"
Jul 13 18:58:44.958343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.958217    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="706f4a9c-fc47-4706-b956-014598801788" containerName="extract"
Jul 13 18:58:44.958343 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.958256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="706f4a9c-fc47-4706-b956-014598801788" containerName="extract"
Jul 13 18:58:44.965416 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.965395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:44.967517 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.967489    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 13 18:58:44.967517 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.967509    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 13 18:58:44.967687 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.967543    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6cwfd\""
Jul 13 18:58:44.974125 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:44.974097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"]
Jul 13 18:58:45.045319 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.045287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.045319 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.045330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.045548 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.045414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6v2\" (UniqueName: \"kubernetes.io/projected/58e23243-29c6-4b3d-b3dc-3667fac7e960-kube-api-access-rs6v2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.146636 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.146598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.146636 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.146640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.146931 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.146683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6v2\" (UniqueName: \"kubernetes.io/projected/58e23243-29c6-4b3d-b3dc-3667fac7e960-kube-api-access-rs6v2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.147149 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.147129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.147229 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.147131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.155345 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.155317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6v2\" (UniqueName: \"kubernetes.io/projected/58e23243-29c6-4b3d-b3dc-3667fac7e960-kube-api-access-rs6v2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.274528 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.274429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:45.403748 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.403718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"]
Jul 13 18:58:45.406859 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:58:45.406831    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e23243_29c6_4b3d_b3dc_3667fac7e960.slice/crio-78ce3eb7a0518b744b901977930fe9d989b09eaae15eb9bbc783bd10e4ce1b0a WatchSource:0}: Error finding container 78ce3eb7a0518b744b901977930fe9d989b09eaae15eb9bbc783bd10e4ce1b0a: Status 404 returned error can't find the container with id 78ce3eb7a0518b744b901977930fe9d989b09eaae15eb9bbc783bd10e4ce1b0a
Jul 13 18:58:45.547912 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.547826    2581 generic.go:356] "Generic (PLEG): container finished" podID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerID="97ab1319fd785d430f8d7e850e0d725d590aac9cf423e371414bc5a5f5356167" exitCode=0
Jul 13 18:58:45.547912 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.547896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9" event={"ID":"58e23243-29c6-4b3d-b3dc-3667fac7e960","Type":"ContainerDied","Data":"97ab1319fd785d430f8d7e850e0d725d590aac9cf423e371414bc5a5f5356167"}
Jul 13 18:58:45.548081 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:45.547936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9" event={"ID":"58e23243-29c6-4b3d-b3dc-3667fac7e960","Type":"ContainerStarted","Data":"78ce3eb7a0518b744b901977930fe9d989b09eaae15eb9bbc783bd10e4ce1b0a"}
Jul 13 18:58:47.159641 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:47.159607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:47.273488 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:47.273441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:48.563262 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:48.563225    2581 generic.go:356] "Generic (PLEG): container finished" podID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerID="22a3f973eb4923266d17d50ed55962f4cdc800691bebfcfd4c5c4ae71887baf5" exitCode=0
Jul 13 18:58:48.563688 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:48.563305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9" event={"ID":"58e23243-29c6-4b3d-b3dc-3667fac7e960","Type":"ContainerDied","Data":"22a3f973eb4923266d17d50ed55962f4cdc800691bebfcfd4c5c4ae71887baf5"}
Jul 13 18:58:49.569295 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:49.569262    2581 generic.go:356] "Generic (PLEG): container finished" podID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerID="873c9a00281a5a58dfb21e35767abfa867d2fd9116dc68845c4f16f58eb11103" exitCode=0
Jul 13 18:58:49.569295 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:49.569299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9" event={"ID":"58e23243-29c6-4b3d-b3dc-3667fac7e960","Type":"ContainerDied","Data":"873c9a00281a5a58dfb21e35767abfa867d2fd9116dc68845c4f16f58eb11103"}
Jul 13 18:58:50.279772 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.279743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:50.702259 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.702235    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:50.792745 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.792712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-bundle\") pod \"58e23243-29c6-4b3d-b3dc-3667fac7e960\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") "
Jul 13 18:58:50.792745 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.792748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-util\") pod \"58e23243-29c6-4b3d-b3dc-3667fac7e960\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") "
Jul 13 18:58:50.793023 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.792774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs6v2\" (UniqueName: \"kubernetes.io/projected/58e23243-29c6-4b3d-b3dc-3667fac7e960-kube-api-access-rs6v2\") pod \"58e23243-29c6-4b3d-b3dc-3667fac7e960\" (UID: \"58e23243-29c6-4b3d-b3dc-3667fac7e960\") "
Jul 13 18:58:50.793230 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.793197    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-bundle" (OuterVolumeSpecName: "bundle") pod "58e23243-29c6-4b3d-b3dc-3667fac7e960" (UID: "58e23243-29c6-4b3d-b3dc-3667fac7e960"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:58:50.795066 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.795039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e23243-29c6-4b3d-b3dc-3667fac7e960-kube-api-access-rs6v2" (OuterVolumeSpecName: "kube-api-access-rs6v2") pod "58e23243-29c6-4b3d-b3dc-3667fac7e960" (UID: "58e23243-29c6-4b3d-b3dc-3667fac7e960"). InnerVolumeSpecName "kube-api-access-rs6v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:58:50.797716 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.797673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-util" (OuterVolumeSpecName: "util") pod "58e23243-29c6-4b3d-b3dc-3667fac7e960" (UID: "58e23243-29c6-4b3d-b3dc-3667fac7e960"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:58:50.894053 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.893955    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-bundle\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:58:50.894053 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.893998    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58e23243-29c6-4b3d-b3dc-3667fac7e960-util\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:58:50.894053 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:50.894007    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6v2\" (UniqueName: \"kubernetes.io/projected/58e23243-29c6-4b3d-b3dc-3667fac7e960-kube-api-access-rs6v2\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:58:51.578196 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:51.578164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9" event={"ID":"58e23243-29c6-4b3d-b3dc-3667fac7e960","Type":"ContainerDied","Data":"78ce3eb7a0518b744b901977930fe9d989b09eaae15eb9bbc783bd10e4ce1b0a"}
Jul 13 18:58:51.578384 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:51.578218    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78ce3eb7a0518b744b901977930fe9d989b09eaae15eb9bbc783bd10e4ce1b0a"
Jul 13 18:58:51.578384 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:51.578169    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg2t9"
Jul 13 18:58:55.729659 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:55.729617    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:58:55.828868 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:58:55.828830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:07.628356 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:07.628320    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:12.947124 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.947089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:12.954387 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954353    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"]
Jul 13 18:59:12.954665 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954647    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerName="extract"
Jul 13 18:59:12.954735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954668    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerName="extract"
Jul 13 18:59:12.954735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954680    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerName="util"
Jul 13 18:59:12.954735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954688    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerName="util"
Jul 13 18:59:12.954735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954697    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerName="pull"
Jul 13 18:59:12.954735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954704    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerName="pull"
Jul 13 18:59:12.955013 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.954764    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e23243-29c6-4b3d-b3dc-3667fac7e960" containerName="extract"
Jul 13 18:59:12.960411 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.960391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:12.962476 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.962448    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 18:59:12.962597 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.962470    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 13 18:59:12.962597 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.962553    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 13 18:59:12.962597 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.962587    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 18:59:12.962825 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.962807    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 13 18:59:12.962896 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.962810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mzqbp\""
Jul 13 18:59:12.962896 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.962809    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 13 18:59:12.970498 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:12.970475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"]
Jul 13 18:59:13.046074 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.046035    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:13.060601 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.060575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.060690 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.060604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.060690 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.060623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9546522-0517-4e48-98d3-5a452af57b10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.060690 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.060639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqhz\" (UniqueName: \"kubernetes.io/projected/f9546522-0517-4e48-98d3-5a452af57b10-kube-api-access-4pqhz\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.060848 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.060682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9546522-0517-4e48-98d3-5a452af57b10-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.060848 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.060715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9546522-0517-4e48-98d3-5a452af57b10-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.060848 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.060746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.161223 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.161181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.161408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.161282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.161408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.161309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.161408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.161334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9546522-0517-4e48-98d3-5a452af57b10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.161408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.161359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqhz\" (UniqueName: \"kubernetes.io/projected/f9546522-0517-4e48-98d3-5a452af57b10-kube-api-access-4pqhz\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.161408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.161383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9546522-0517-4e48-98d3-5a452af57b10-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.161408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.161404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9546522-0517-4e48-98d3-5a452af57b10-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.163709 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.163673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9546522-0517-4e48-98d3-5a452af57b10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.164699 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.164668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.164837 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.164720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.168377 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.168351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9546522-0517-4e48-98d3-5a452af57b10-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.168485 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.168430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9546522-0517-4e48-98d3-5a452af57b10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.171099 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.171051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqhz\" (UniqueName: \"kubernetes.io/projected/f9546522-0517-4e48-98d3-5a452af57b10-kube-api-access-4pqhz\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.173016 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.172995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9546522-0517-4e48-98d3-5a452af57b10-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zprl6\" (UID: \"f9546522-0517-4e48-98d3-5a452af57b10\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.270398 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.270369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:13.412105 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.411978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"]
Jul 13 18:59:13.414710 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:13.414684    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9546522_0517_4e48_98d3_5a452af57b10.slice/crio-a25b6b0c58fad791149a7e9247e747733e14d89ef1ef7932ef106fd0ba79d09a WatchSource:0}: Error finding container a25b6b0c58fad791149a7e9247e747733e14d89ef1ef7932ef106fd0ba79d09a: Status 404 returned error can't find the container with id a25b6b0c58fad791149a7e9247e747733e14d89ef1ef7932ef106fd0ba79d09a
Jul 13 18:59:13.653606 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.653523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6" event={"ID":"f9546522-0517-4e48-98d3-5a452af57b10","Type":"ContainerStarted","Data":"a25b6b0c58fad791149a7e9247e747733e14d89ef1ef7932ef106fd0ba79d09a"}
Jul 13 18:59:13.939687 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:13.939603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:15.953319 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:15.953279    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 18:59:15.953544 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:15.953368    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 18:59:16.523022 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:16.522931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:16.579472 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:16.579385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:16.668654 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:16.668619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6" event={"ID":"f9546522-0517-4e48-98d3-5a452af57b10","Type":"ContainerStarted","Data":"65bfdc2ab283855694ab8840d9962a235d1c2572eaf03523dd624b213bd941e9"}
Jul 13 18:59:16.668853 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:16.668752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:16.692559 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:16.692500    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6" podStartSLOduration=2.156107224 podStartE2EDuration="4.692482955s" totalImagesPullingTime="2.536375731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:16.690631028 +0000 UTC m=+593.590856420" watchObservedRunningTime="2026-07-13 18:59:16.692482955 +0000 UTC m=+593.592708348"
Jul 13 18:59:17.675025 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:17.674995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zprl6"
Jul 13 18:59:19.335502 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:19.335463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:19.449875 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:19.449840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:22.486452 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:22.486411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:23.541069 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:23.541044    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:59:23.541751 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:23.541727    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 18:59:25.798883 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:25.798851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:25.899484 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:25.899451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:26.120532 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:26.120452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:26.220836 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:26.220799    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:28.964364 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.964329    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"]
Jul 13 18:59:28.968891 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.968874    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:28.971145 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.971124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 13 18:59:28.971734 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.971708    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 13 18:59:28.971835 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.971741    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6cwfd\""
Jul 13 18:59:28.977045 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.977017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"]
Jul 13 18:59:28.989517 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.989489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:28.989642 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.989532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:28.989642 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:28.989559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45std\" (UniqueName: \"kubernetes.io/projected/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-kube-api-access-45std\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.056449 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.056412    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"]
Jul 13 18:59:29.059719 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.059696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.068416 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.068388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"]
Jul 13 18:59:29.089976 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.089949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45std\" (UniqueName: \"kubernetes.io/projected/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-kube-api-access-45std\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.090125 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.089985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.090125 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.090018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.090125 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.090036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.090260 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.090178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.090260 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.090215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx2g\" (UniqueName: \"kubernetes.io/projected/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-kube-api-access-gkx2g\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.090373 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.090359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.090409 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.090397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.098066 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.098040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45std\" (UniqueName: \"kubernetes.io/projected/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-kube-api-access-45std\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.155677 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.155647    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"]
Jul 13 18:59:29.159070 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.159054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.168244 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.168219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"]
Jul 13 18:59:29.191176 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9tv\" (UniqueName: \"kubernetes.io/projected/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-kube-api-access-ql9tv\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.191313 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.191313 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx2g\" (UniqueName: \"kubernetes.io/projected/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-kube-api-access-gkx2g\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.191313 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.191411 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.191411 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.191583 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.191655 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.191640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.198761 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.198735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx2g\" (UniqueName: \"kubernetes.io/projected/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-kube-api-access-gkx2g\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.259765 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.259703    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"]
Jul 13 18:59:29.262939 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.262923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.273973 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.273951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"]
Jul 13 18:59:29.278481 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.278461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:29.294179 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.294147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.294362 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.294342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.294488 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.294473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.294601 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.294587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.294713 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.294693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkt5g\" (UniqueName: \"kubernetes.io/projected/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-kube-api-access-rkt5g\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.294858 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.294833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9tv\" (UniqueName: \"kubernetes.io/projected/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-kube-api-access-ql9tv\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.295716 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.295693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.296253 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.296233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.304390 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.304368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9tv\" (UniqueName: \"kubernetes.io/projected/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-kube-api-access-ql9tv\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.369224 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.369184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:29.396032 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.395996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.396206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.396051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkt5g\" (UniqueName: \"kubernetes.io/projected/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-kube-api-access-rkt5g\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.396206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.396173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.396455 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.396432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.396509 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.396490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.405607 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.405572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkt5g\" (UniqueName: \"kubernetes.io/projected/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-kube-api-access-rkt5g\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.426971 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.426665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"]
Jul 13 18:59:29.430106 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:29.430067    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e202fe9_fce5_47b8_ba8a_f99babf1c4c6.slice/crio-e42eaf87f7815eae4184b7d3033eb2f58d1e130adbe0aa009a67c125123275df WatchSource:0}: Error finding container e42eaf87f7815eae4184b7d3033eb2f58d1e130adbe0aa009a67c125123275df: Status 404 returned error can't find the container with id e42eaf87f7815eae4184b7d3033eb2f58d1e130adbe0aa009a67c125123275df
Jul 13 18:59:29.468051 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.467949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:29.516254 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.516228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"]
Jul 13 18:59:29.518074 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:29.518039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2961e1a_9e74_41c2_b2ba_194ef02cd2f4.slice/crio-a015e3f153821b8413e2bc086c4c259677c5618665b3c1bd85e813fd9473e7ef WatchSource:0}: Error finding container a015e3f153821b8413e2bc086c4c259677c5618665b3c1bd85e813fd9473e7ef: Status 404 returned error can't find the container with id a015e3f153821b8413e2bc086c4c259677c5618665b3c1bd85e813fd9473e7ef
Jul 13 18:59:29.576119 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.572043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:29.627206 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.626336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"]
Jul 13 18:59:29.632339 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:29.632270    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9249f2_4eb1_40bf_a3e1_1bdbef14a33e.slice/crio-1cc419170a91ed1eb0a3c6bbbf52761b7852e90390d2e5489958bc06c607aa75 WatchSource:0}: Error finding container 1cc419170a91ed1eb0a3c6bbbf52761b7852e90390d2e5489958bc06c607aa75: Status 404 returned error can't find the container with id 1cc419170a91ed1eb0a3c6bbbf52761b7852e90390d2e5489958bc06c607aa75
Jul 13 18:59:29.718955 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.718918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf" event={"ID":"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4","Type":"ContainerStarted","Data":"9971d86ec9650a575fe0309a2c9d7037137273f8ed24008663bc071e9fbde8a3"}
Jul 13 18:59:29.719096 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.718989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf" event={"ID":"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4","Type":"ContainerStarted","Data":"a015e3f153821b8413e2bc086c4c259677c5618665b3c1bd85e813fd9473e7ef"}
Jul 13 18:59:29.720536 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.720508    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerID="b3eb0fc8a1472241b0297ea9490497186bfb823a7820a794aff7f94a9a7e0819" exitCode=0
Jul 13 18:59:29.720661 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.720585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f" event={"ID":"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6","Type":"ContainerDied","Data":"b3eb0fc8a1472241b0297ea9490497186bfb823a7820a794aff7f94a9a7e0819"}
Jul 13 18:59:29.720661 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.720634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f" event={"ID":"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6","Type":"ContainerStarted","Data":"e42eaf87f7815eae4184b7d3033eb2f58d1e130adbe0aa009a67c125123275df"}
Jul 13 18:59:29.723412 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.723315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc" event={"ID":"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e","Type":"ContainerStarted","Data":"6e76db2bbc0e39a9d4efd0cec88b7a13c82a19c6b77b432b7a34d48aef80efb2"}
Jul 13 18:59:29.723412 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.723379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc" event={"ID":"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e","Type":"ContainerStarted","Data":"1cc419170a91ed1eb0a3c6bbbf52761b7852e90390d2e5489958bc06c607aa75"}
Jul 13 18:59:29.725309 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:29.725284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"]
Jul 13 18:59:29.774838 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:29.774736    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a7bfdb_e6cb_4a75_9a5d_1cd1977900cc.slice/crio-129a92e1c1427b9fe1e785a33c0ca9b26e7d351c5b44a990904505521edd098d WatchSource:0}: Error finding container 129a92e1c1427b9fe1e785a33c0ca9b26e7d351c5b44a990904505521edd098d: Status 404 returned error can't find the container with id 129a92e1c1427b9fe1e785a33c0ca9b26e7d351c5b44a990904505521edd098d
Jul 13 18:59:30.729825 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:30.729777    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerID="6e76db2bbc0e39a9d4efd0cec88b7a13c82a19c6b77b432b7a34d48aef80efb2" exitCode=0
Jul 13 18:59:30.729825 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:30.729822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc" event={"ID":"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e","Type":"ContainerDied","Data":"6e76db2bbc0e39a9d4efd0cec88b7a13c82a19c6b77b432b7a34d48aef80efb2"}
Jul 13 18:59:30.731258 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:30.731232    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerID="9b97d915fd90be38af0e49b72b4650cfac5bb4d956fd29171597291d6be67eb3" exitCode=0
Jul 13 18:59:30.731362 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:30.731298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x" event={"ID":"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc","Type":"ContainerDied","Data":"9b97d915fd90be38af0e49b72b4650cfac5bb4d956fd29171597291d6be67eb3"}
Jul 13 18:59:30.731362 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:30.731328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x" event={"ID":"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc","Type":"ContainerStarted","Data":"129a92e1c1427b9fe1e785a33c0ca9b26e7d351c5b44a990904505521edd098d"}
Jul 13 18:59:30.732728 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:30.732710    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerID="9971d86ec9650a575fe0309a2c9d7037137273f8ed24008663bc071e9fbde8a3" exitCode=0
Jul 13 18:59:30.732803 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:30.732765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf" event={"ID":"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4","Type":"ContainerDied","Data":"9971d86ec9650a575fe0309a2c9d7037137273f8ed24008663bc071e9fbde8a3"}
Jul 13 18:59:31.739746 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:31.739719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc" event={"ID":"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e","Type":"ContainerStarted","Data":"db93fe281bca89c6733a9916def6920952d323cb1e17edce6d6775a167d3b47d"}
Jul 13 18:59:31.741753 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:31.741726    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerID="33de14c0ed09e9dc491cf713fefeea7951c93c671b140d919dcdc868a4896105" exitCode=0
Jul 13 18:59:31.742051 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:31.741814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x" event={"ID":"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc","Type":"ContainerDied","Data":"33de14c0ed09e9dc491cf713fefeea7951c93c671b140d919dcdc868a4896105"}
Jul 13 18:59:31.750627 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:31.750599    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerID="d8ea8e714ac46ff04ae065d57c8e425f43baaae9e56946317784b7a360103447" exitCode=0
Jul 13 18:59:31.750782 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:31.750717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf" event={"ID":"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4","Type":"ContainerDied","Data":"d8ea8e714ac46ff04ae065d57c8e425f43baaae9e56946317784b7a360103447"}
Jul 13 18:59:31.753025 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:31.752998    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerID="8650c69004dc8f43d7b7246375d0c3476e5cd58325b8afe218758de0946f6fb7" exitCode=0
Jul 13 18:59:31.753106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:31.753040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f" event={"ID":"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6","Type":"ContainerDied","Data":"8650c69004dc8f43d7b7246375d0c3476e5cd58325b8afe218758de0946f6fb7"}
Jul 13 18:59:32.759730 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.759697    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerID="c5ccad36d09c8a50ba793a07119f8ce6030b347d7aa872f009f75c062c936171" exitCode=0
Jul 13 18:59:32.760184 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.759776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f" event={"ID":"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6","Type":"ContainerDied","Data":"c5ccad36d09c8a50ba793a07119f8ce6030b347d7aa872f009f75c062c936171"}
Jul 13 18:59:32.761805 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.761761    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerID="db93fe281bca89c6733a9916def6920952d323cb1e17edce6d6775a167d3b47d" exitCode=0
Jul 13 18:59:32.761805 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.761781    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerID="bf4a261373bf00657c91914fd82091f881265e01f4747da89c6455227584a6bc" exitCode=0
Jul 13 18:59:32.761964 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.761826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc" event={"ID":"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e","Type":"ContainerDied","Data":"db93fe281bca89c6733a9916def6920952d323cb1e17edce6d6775a167d3b47d"}
Jul 13 18:59:32.761964 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.761869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc" event={"ID":"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e","Type":"ContainerDied","Data":"bf4a261373bf00657c91914fd82091f881265e01f4747da89c6455227584a6bc"}
Jul 13 18:59:32.764029 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.764006    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerID="05dd3473d7d55a636f10cc6a4e168e92409655d6c385275ab86b07788ed2e234" exitCode=0
Jul 13 18:59:32.764140 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.764087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x" event={"ID":"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc","Type":"ContainerDied","Data":"05dd3473d7d55a636f10cc6a4e168e92409655d6c385275ab86b07788ed2e234"}
Jul 13 18:59:32.766001 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.765981    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerID="23c03a3d6e3e71a1cde670f030af7b3ea3c19d3d6dfa930c16810735a23fd5b2" exitCode=0
Jul 13 18:59:32.766109 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:32.766009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf" event={"ID":"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4","Type":"ContainerDied","Data":"23c03a3d6e3e71a1cde670f030af7b3ea3c19d3d6dfa930c16810735a23fd5b2"}
Jul 13 18:59:33.918452 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.918430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:33.930869 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.930844    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-util\") pod \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") "
Jul 13 18:59:33.930987 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.930934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-bundle\") pod \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") "
Jul 13 18:59:33.930987 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.930968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45std\" (UniqueName: \"kubernetes.io/projected/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-kube-api-access-45std\") pod \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\" (UID: \"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6\") "
Jul 13 18:59:33.931685 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.931652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-bundle" (OuterVolumeSpecName: "bundle") pod "1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" (UID: "1e202fe9-fce5-47b8-ba8a-f99babf1c4c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:33.931894 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.931874    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:33.933532 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.933498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-kube-api-access-45std" (OuterVolumeSpecName: "kube-api-access-45std") pod "1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" (UID: "1e202fe9-fce5-47b8-ba8a-f99babf1c4c6"). InnerVolumeSpecName "kube-api-access-45std". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:59:33.936379 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.936355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-util" (OuterVolumeSpecName: "util") pod "1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" (UID: "1e202fe9-fce5-47b8-ba8a-f99babf1c4c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:33.975544 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.975515    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:33.977980 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:33.977961    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:34.032363 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-util\") pod \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") "
Jul 13 18:59:34.032363 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032322    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-bundle\") pod \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") "
Jul 13 18:59:34.032363 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032356    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql9tv\" (UniqueName: \"kubernetes.io/projected/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-kube-api-access-ql9tv\") pod \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") "
Jul 13 18:59:34.032637 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032390    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-util\") pod \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") "
Jul 13 18:59:34.032637 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032419    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkx2g\" (UniqueName: \"kubernetes.io/projected/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-kube-api-access-gkx2g\") pod \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") "
Jul 13 18:59:34.032637 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032461    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-util\") pod \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\" (UID: \"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e\") "
Jul 13 18:59:34.032637 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkt5g\" (UniqueName: \"kubernetes.io/projected/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-kube-api-access-rkt5g\") pod \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") "
Jul 13 18:59:34.032637 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-bundle\") pod \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\" (UID: \"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4\") "
Jul 13 18:59:34.032637 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-bundle\") pod \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\" (UID: \"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc\") "
Jul 13 18:59:34.032963 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032754    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-util\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.032963 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032768    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-bundle\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.032963 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.032781    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45std\" (UniqueName: \"kubernetes.io/projected/1e202fe9-fce5-47b8-ba8a-f99babf1c4c6-kube-api-access-45std\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.033287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.033209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-bundle" (OuterVolumeSpecName: "bundle") pod "b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" (UID: "b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:34.033373 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.033340    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-bundle" (OuterVolumeSpecName: "bundle") pod "7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" (UID: "7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:34.033770 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.033747    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-bundle" (OuterVolumeSpecName: "bundle") pod "f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" (UID: "f2961e1a-9e74-41c2-b2ba-194ef02cd2f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:34.034888 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.034858    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-kube-api-access-ql9tv" (OuterVolumeSpecName: "kube-api-access-ql9tv") pod "7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" (UID: "7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e"). InnerVolumeSpecName "kube-api-access-ql9tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:59:34.035300 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.035271    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-kube-api-access-gkx2g" (OuterVolumeSpecName: "kube-api-access-gkx2g") pod "f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" (UID: "f2961e1a-9e74-41c2-b2ba-194ef02cd2f4"). InnerVolumeSpecName "kube-api-access-gkx2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:59:34.035577 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.035561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-kube-api-access-rkt5g" (OuterVolumeSpecName: "kube-api-access-rkt5g") pod "b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" (UID: "b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc"). InnerVolumeSpecName "kube-api-access-rkt5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:59:34.038997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.038965    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-util" (OuterVolumeSpecName: "util") pod "f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" (UID: "f2961e1a-9e74-41c2-b2ba-194ef02cd2f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:34.041364 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.041341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-util" (OuterVolumeSpecName: "util") pod "b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" (UID: "b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:34.041713 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.041697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-util" (OuterVolumeSpecName: "util") pod "7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" (UID: "7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:34.133602 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133563    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9tv\" (UniqueName: \"kubernetes.io/projected/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-kube-api-access-ql9tv\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133602 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133597    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-util\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133602 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133609    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkx2g\" (UniqueName: \"kubernetes.io/projected/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-kube-api-access-gkx2g\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133877 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133620    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-util\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133877 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133632    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkt5g\" (UniqueName: \"kubernetes.io/projected/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-kube-api-access-rkt5g\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133877 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133642    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-bundle\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133877 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133653    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc-bundle\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133877 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133662    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2961e1a-9e74-41c2-b2ba-194ef02cd2f4-util\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.133877 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.133673    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e-bundle\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 18:59:34.776231 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.776200    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf"
Jul 13 18:59:34.776404 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.776223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36twgrf" event={"ID":"f2961e1a-9e74-41c2-b2ba-194ef02cd2f4","Type":"ContainerDied","Data":"a015e3f153821b8413e2bc086c4c259677c5618665b3c1bd85e813fd9473e7ef"}
Jul 13 18:59:34.776404 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.776277    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a015e3f153821b8413e2bc086c4c259677c5618665b3c1bd85e813fd9473e7ef"
Jul 13 18:59:34.778121 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.778097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f" event={"ID":"1e202fe9-fce5-47b8-ba8a-f99babf1c4c6","Type":"ContainerDied","Data":"e42eaf87f7815eae4184b7d3033eb2f58d1e130adbe0aa009a67c125123275df"}
Jul 13 18:59:34.778248 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.778123    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabvmn9f"
Jul 13 18:59:34.778248 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.778132    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42eaf87f7815eae4184b7d3033eb2f58d1e130adbe0aa009a67c125123275df"
Jul 13 18:59:34.780067 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.780052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc"
Jul 13 18:59:34.780163 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.780079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76hn9vc" event={"ID":"7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e","Type":"ContainerDied","Data":"1cc419170a91ed1eb0a3c6bbbf52761b7852e90390d2e5489958bc06c607aa75"}
Jul 13 18:59:34.780163 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.780121    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc419170a91ed1eb0a3c6bbbf52761b7852e90390d2e5489958bc06c607aa75"
Jul 13 18:59:34.781928 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.781908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x" event={"ID":"b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc","Type":"ContainerDied","Data":"129a92e1c1427b9fe1e785a33c0ca9b26e7d351c5b44a990904505521edd098d"}
Jul 13 18:59:34.782019 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.781940    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="129a92e1c1427b9fe1e785a33c0ca9b26e7d351c5b44a990904505521edd098d"
Jul 13 18:59:34.782019 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:34.781943    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efwwd2x"
Jul 13 18:59:40.374127 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:40.374093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:43.180639 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180610    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"]
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180868    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180878    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180887    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180898    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180904    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180910    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180925    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180930    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerName="util"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180936    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180940    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180947    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerName="pull"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180952    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerName="pull"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180958    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerName="pull"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180963    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerName="pull"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180974    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180978    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180983    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180988    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerName="extract"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.180995    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerName="pull"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.181000    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerName="pull"
Jul 13 18:59:43.180997 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.181007    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerName="pull"
Jul 13 18:59:43.181617 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.181012    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerName="pull"
Jul 13 18:59:43.181617 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.181049    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9249f2-4eb1-40bf-a3e1-1bdbef14a33e" containerName="extract"
Jul 13 18:59:43.181617 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.181056    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2961e1a-9e74-41c2-b2ba-194ef02cd2f4" containerName="extract"
Jul 13 18:59:43.181617 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.181062    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e202fe9-fce5-47b8-ba8a-f99babf1c4c6" containerName="extract"
Jul 13 18:59:43.181617 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.181068    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a7bfdb-e6cb-4a75-9a5d-1cd1977900cc" containerName="extract"
Jul 13 18:59:43.185232 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.185215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"
Jul 13 18:59:43.187616 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.187592    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 13 18:59:43.187745 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.187675    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 13 18:59:43.188124 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.188109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gcknn\""
Jul 13 18:59:43.200653 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.200616    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"]
Jul 13 18:59:43.297078 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.297049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch799\" (UniqueName: \"kubernetes.io/projected/75351e08-fdf3-40d9-a685-e3d447b251c6-kube-api-access-ch799\") pod \"limitador-operator-controller-manager-6f9f468797-mw8p8\" (UID: \"75351e08-fdf3-40d9-a685-e3d447b251c6\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"
Jul 13 18:59:43.398276 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.398242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch799\" (UniqueName: \"kubernetes.io/projected/75351e08-fdf3-40d9-a685-e3d447b251c6-kube-api-access-ch799\") pod \"limitador-operator-controller-manager-6f9f468797-mw8p8\" (UID: \"75351e08-fdf3-40d9-a685-e3d447b251c6\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"
Jul 13 18:59:43.411723 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.411695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch799\" (UniqueName: \"kubernetes.io/projected/75351e08-fdf3-40d9-a685-e3d447b251c6-kube-api-access-ch799\") pod \"limitador-operator-controller-manager-6f9f468797-mw8p8\" (UID: \"75351e08-fdf3-40d9-a685-e3d447b251c6\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"
Jul 13 18:59:43.494807 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.494765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"
Jul 13 18:59:43.640972 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.640936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"]
Jul 13 18:59:43.644117 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:43.644087    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75351e08_fdf3_40d9_a685_e3d447b251c6.slice/crio-8a3ebec927fe960381989343c80a1a9a9bf82558af880407ded68b57f50fc408 WatchSource:0}: Error finding container 8a3ebec927fe960381989343c80a1a9a9bf82558af880407ded68b57f50fc408: Status 404 returned error can't find the container with id 8a3ebec927fe960381989343c80a1a9a9bf82558af880407ded68b57f50fc408
Jul 13 18:59:43.820901 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.820811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8" event={"ID":"75351e08-fdf3-40d9-a685-e3d447b251c6","Type":"ContainerStarted","Data":"8a3ebec927fe960381989343c80a1a9a9bf82558af880407ded68b57f50fc408"}
Jul 13 18:59:43.916319 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:43.916288    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:44.527515 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:44.527478    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:44.934380 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:44.934298    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:46.836582 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:46.836544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8" event={"ID":"75351e08-fdf3-40d9-a685-e3d447b251c6","Type":"ContainerStarted","Data":"0a9d3e172de5348d6d022f58a5164f7befa22d0fcd3916d204041a28f4d69340"}
Jul 13 18:59:46.837056 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:46.836685    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"
Jul 13 18:59:46.856362 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:46.856311    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8" podStartSLOduration=1.1413696469999999 podStartE2EDuration="3.856295667s" totalImagesPullingTime="2.71492602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:46.854605543 +0000 UTC m=+623.754830936" watchObservedRunningTime="2026-07-13 18:59:46.856295667 +0000 UTC m=+623.756521058"
Jul 13 18:59:47.724946 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.724916    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"]
Jul 13 18:59:47.733477 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.733452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:47.736008 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.735984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k2k88\""
Jul 13 18:59:47.743379 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.743358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"]
Jul 13 18:59:47.833669 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.833629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74x6\" (UniqueName: \"kubernetes.io/projected/e7970fea-e3da-4538-88a3-dda7d391b794-kube-api-access-l74x6\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-cf52x\" (UID: \"e7970fea-e3da-4538-88a3-dda7d391b794\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:47.833669 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.833671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7970fea-e3da-4538-88a3-dda7d391b794-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-cf52x\" (UID: \"e7970fea-e3da-4538-88a3-dda7d391b794\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:47.914281 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.914250    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:47.934298 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.934273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74x6\" (UniqueName: \"kubernetes.io/projected/e7970fea-e3da-4538-88a3-dda7d391b794-kube-api-access-l74x6\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-cf52x\" (UID: \"e7970fea-e3da-4538-88a3-dda7d391b794\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:47.934408 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.934304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7970fea-e3da-4538-88a3-dda7d391b794-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-cf52x\" (UID: \"e7970fea-e3da-4538-88a3-dda7d391b794\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:47.934646 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.934629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7970fea-e3da-4538-88a3-dda7d391b794-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-cf52x\" (UID: \"e7970fea-e3da-4538-88a3-dda7d391b794\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:47.943228 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:47.943204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74x6\" (UniqueName: \"kubernetes.io/projected/e7970fea-e3da-4538-88a3-dda7d391b794-kube-api-access-l74x6\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-cf52x\" (UID: \"e7970fea-e3da-4538-88a3-dda7d391b794\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:48.044430 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:48.044342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:48.195568 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:48.195437    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"]
Jul 13 18:59:48.198417 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:48.198390    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7970fea_e3da_4538_88a3_dda7d391b794.slice/crio-a022183e6cedb856e4ced345f2ca9a7217f59fe248bec2a726afa22f88fe709c WatchSource:0}: Error finding container a022183e6cedb856e4ced345f2ca9a7217f59fe248bec2a726afa22f88fe709c: Status 404 returned error can't find the container with id a022183e6cedb856e4ced345f2ca9a7217f59fe248bec2a726afa22f88fe709c
Jul 13 18:59:48.316036 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:48.315954    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:48.417963 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:48.417917    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:48.846992 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:48.846956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x" event={"ID":"e7970fea-e3da-4538-88a3-dda7d391b794","Type":"ContainerStarted","Data":"a022183e6cedb856e4ced345f2ca9a7217f59fe248bec2a726afa22f88fe709c"}
Jul 13 18:59:49.239699 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:49.239659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:49.391671 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:49.391640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:50.788345 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:50.788312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:51.025735 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:51.025700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:52.168882 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.168848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:52.684575 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.684544    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"]
Jul 13 18:59:52.689049 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.689026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"
Jul 13 18:59:52.692364 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.692343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 13 18:59:52.692860 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.692840    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q567v\""
Jul 13 18:59:52.715117 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.715088    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"]
Jul 13 18:59:52.778757 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.778713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpz7\" (UniqueName: \"kubernetes.io/projected/03ff69ad-9d85-4022-a848-d302e2770e6c-kube-api-access-blpz7\") pod \"dns-operator-controller-manager-65b49595d7-hphfv\" (UID: \"03ff69ad-9d85-4022-a848-d302e2770e6c\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"
Jul 13 18:59:52.879683 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.879651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpz7\" (UniqueName: \"kubernetes.io/projected/03ff69ad-9d85-4022-a848-d302e2770e6c-kube-api-access-blpz7\") pod \"dns-operator-controller-manager-65b49595d7-hphfv\" (UID: \"03ff69ad-9d85-4022-a848-d302e2770e6c\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"
Jul 13 18:59:52.888084 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:52.888056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpz7\" (UniqueName: \"kubernetes.io/projected/03ff69ad-9d85-4022-a848-d302e2770e6c-kube-api-access-blpz7\") pod \"dns-operator-controller-manager-65b49595d7-hphfv\" (UID: \"03ff69ad-9d85-4022-a848-d302e2770e6c\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"
Jul 13 18:59:53.001528 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:53.001485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"
Jul 13 18:59:53.134071 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:53.134048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"]
Jul 13 18:59:53.136401 ip-10-0-131-203 kubenswrapper[2581]: W0713 18:59:53.136362    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ff69ad_9d85_4022_a848_d302e2770e6c.slice/crio-e7b7fbd898dc00772607792416f012cb9abf7ee4b0fd5739a43ee27aa79a6c55 WatchSource:0}: Error finding container e7b7fbd898dc00772607792416f012cb9abf7ee4b0fd5739a43ee27aa79a6c55: Status 404 returned error can't find the container with id e7b7fbd898dc00772607792416f012cb9abf7ee4b0fd5739a43ee27aa79a6c55
Jul 13 18:59:53.874461 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:53.874421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv" event={"ID":"03ff69ad-9d85-4022-a848-d302e2770e6c","Type":"ContainerStarted","Data":"e7b7fbd898dc00772607792416f012cb9abf7ee4b0fd5739a43ee27aa79a6c55"}
Jul 13 18:59:53.876134 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:53.876108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x" event={"ID":"e7970fea-e3da-4538-88a3-dda7d391b794","Type":"ContainerStarted","Data":"77adb4d3786ea4f421e3cb05175ce15bf1327e8d9e544a64c34a051ce74786ce"}
Jul 13 18:59:53.876260 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:53.876183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 18:59:53.897601 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:53.897551    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x" podStartSLOduration=2.1788111629999998 podStartE2EDuration="6.897535124s" totalImagesPullingTime="4.718723961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:53.895194913 +0000 UTC m=+630.795420305" watchObservedRunningTime="2026-07-13 18:59:53.897535124 +0000 UTC m=+630.797760515"
Jul 13 18:59:54.147947 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:54.147856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:54.246287 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:54.246254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:55.362840 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:55.362808    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:55.776884 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:55.776848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:55.886920 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:55.886887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv" event={"ID":"03ff69ad-9d85-4022-a848-d302e2770e6c","Type":"ContainerStarted","Data":"84da0e6cb9a9abb1e0edebdd555d7a5da52d703a8a015be759d9b99b369e3e94"}
Jul 13 18:59:55.887100 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:55.887032    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"
Jul 13 18:59:55.904864 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:55.904812    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv" podStartSLOduration=2.118843959 podStartE2EDuration="3.904777377s" totalImagesPullingTime="1.785933418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:55.903250136 +0000 UTC m=+632.803475528" watchObservedRunningTime="2026-07-13 18:59:55.904777377 +0000 UTC m=+632.805002768"
Jul 13 18:59:57.843205 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:57.843174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-mw8p8"
Jul 13 18:59:58.129352 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:58.129251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:58.229736 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:58.229700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:58.527738 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:58.527702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:58.628106 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:58.628076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:58.931448 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:58.931364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:59.060807 ip-10-0-131-203 kubenswrapper[2581]: I0713 18:59:59.060752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:04.882567 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:04.882535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x"
Jul 13 19:00:06.668724 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:06.668686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:06.763197 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:06.763165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:06.893913 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:06.893889    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-hphfv"
Jul 13 19:00:09.043969 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:09.043924    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:09.140070 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:09.140042    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:13.411727 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:13.411689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:13.513385 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:13.513340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:17.241424 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:17.241393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:17.342238 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:17.342201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:17.443979 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:17.443939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:17.497136 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:17.497026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:18.393626 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:18.393592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:18.471659 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:18.471619    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:20.425880 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:20.425844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:20.505056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:20.505015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:27.862567 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:27.862526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:27.962047 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:27.962008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:29.767235 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:29.767168    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:29.863309 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:29.863274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:38.401064 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.400999    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:38.401942 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.401916    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"]
Jul 13 19:00:38.407468 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.407447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.409723 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.409698    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 13 19:00:38.409839 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.409698    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h74mc\""
Jul 13 19:00:38.413590 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.413570    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"]
Jul 13 19:00:38.429708 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.429685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qkl\" (UniqueName: \"kubernetes.io/projected/7df03b34-c12d-4743-bfa7-c49f037b66b2-kube-api-access-d5qkl\") pod \"limitador-limitador-ff68ff5cb-4ph8z\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.429853 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.429738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7df03b34-c12d-4743-bfa7-c49f037b66b2-config-file\") pod \"limitador-limitador-ff68ff5cb-4ph8z\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.499983 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.499952    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"]
Jul 13 19:00:38.506639 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.506609    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:38.530323 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.530296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkl\" (UniqueName: \"kubernetes.io/projected/7df03b34-c12d-4743-bfa7-c49f037b66b2-kube-api-access-d5qkl\") pod \"limitador-limitador-ff68ff5cb-4ph8z\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.530487 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.530352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7df03b34-c12d-4743-bfa7-c49f037b66b2-config-file\") pod \"limitador-limitador-ff68ff5cb-4ph8z\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.530941 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.530923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7df03b34-c12d-4743-bfa7-c49f037b66b2-config-file\") pod \"limitador-limitador-ff68ff5cb-4ph8z\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.538292 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.538269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qkl\" (UniqueName: \"kubernetes.io/projected/7df03b34-c12d-4743-bfa7-c49f037b66b2-kube-api-access-d5qkl\") pod \"limitador-limitador-ff68ff5cb-4ph8z\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.603518 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.603484    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:38.717747 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.717719    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:00:38.882604 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:38.882572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"]
Jul 13 19:00:38.886245 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:00:38.886220    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df03b34_c12d_4743_bfa7_c49f037b66b2.slice/crio-ea5977599a6e28a0b3824fad9e4edf89228d43b8b19addd0e52f39e25c8dcb6a WatchSource:0}: Error finding container ea5977599a6e28a0b3824fad9e4edf89228d43b8b19addd0e52f39e25c8dcb6a: Status 404 returned error can't find the container with id ea5977599a6e28a0b3824fad9e4edf89228d43b8b19addd0e52f39e25c8dcb6a
Jul 13 19:00:39.080324 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:39.080235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z" event={"ID":"7df03b34-c12d-4743-bfa7-c49f037b66b2","Type":"ContainerStarted","Data":"ea5977599a6e28a0b3824fad9e4edf89228d43b8b19addd0e52f39e25c8dcb6a"}
Jul 13 19:00:39.248106 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:39.248074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:41.022526 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:00:41.022491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:08.055464 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:01:08.055432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:14.096755 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:01:14.096723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:25.294949 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:01:25.294914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z" event={"ID":"7df03b34-c12d-4743-bfa7-c49f037b66b2","Type":"ContainerStarted","Data":"4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5"}
Jul 13 19:01:25.295332 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:01:25.295043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:01:25.313869 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:01:25.313816    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z" podStartSLOduration=1.8992084999999999 podStartE2EDuration="47.313783011s" totalImagesPullingTime="45.414574511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:01:25.311154695 +0000 UTC m=+722.211380087" watchObservedRunningTime="2026-07-13 19:01:25.313783011 +0000 UTC m=+722.214008402"
Jul 13 19:01:36.303169 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:01:36.303140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:02:04.497066 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:04.497029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:10.417304 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:10.417265    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:12.443728 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:12.443688    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"]
Jul 13 19:02:12.444167 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:12.444045    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z" podUID="7df03b34-c12d-4743-bfa7-c49f037b66b2" containerName="limitador" containerID="cri-o://4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5" gracePeriod=30
Jul 13 19:02:12.986803 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:12.986771    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:02:13.029205 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.029126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7df03b34-c12d-4743-bfa7-c49f037b66b2-config-file\") pod \"7df03b34-c12d-4743-bfa7-c49f037b66b2\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") "
Jul 13 19:02:13.029205 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.029176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5qkl\" (UniqueName: \"kubernetes.io/projected/7df03b34-c12d-4743-bfa7-c49f037b66b2-kube-api-access-d5qkl\") pod \"7df03b34-c12d-4743-bfa7-c49f037b66b2\" (UID: \"7df03b34-c12d-4743-bfa7-c49f037b66b2\") "
Jul 13 19:02:13.029490 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.029468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df03b34-c12d-4743-bfa7-c49f037b66b2-config-file" (OuterVolumeSpecName: "config-file") pod "7df03b34-c12d-4743-bfa7-c49f037b66b2" (UID: "7df03b34-c12d-4743-bfa7-c49f037b66b2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:02:13.031373 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.031347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df03b34-c12d-4743-bfa7-c49f037b66b2-kube-api-access-d5qkl" (OuterVolumeSpecName: "kube-api-access-d5qkl") pod "7df03b34-c12d-4743-bfa7-c49f037b66b2" (UID: "7df03b34-c12d-4743-bfa7-c49f037b66b2"). InnerVolumeSpecName "kube-api-access-d5qkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:02:13.129622 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.129591    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7df03b34-c12d-4743-bfa7-c49f037b66b2-config-file\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:02:13.129622 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.129619    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qkl\" (UniqueName: \"kubernetes.io/projected/7df03b34-c12d-4743-bfa7-c49f037b66b2-kube-api-access-d5qkl\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:02:13.511940 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.511904    2581 generic.go:356] "Generic (PLEG): container finished" podID="7df03b34-c12d-4743-bfa7-c49f037b66b2" containerID="4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5" exitCode=0
Jul 13 19:02:13.512332 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.511962    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"
Jul 13 19:02:13.512332 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.511975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z" event={"ID":"7df03b34-c12d-4743-bfa7-c49f037b66b2","Type":"ContainerDied","Data":"4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5"}
Jul 13 19:02:13.512332 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.512044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z" event={"ID":"7df03b34-c12d-4743-bfa7-c49f037b66b2","Type":"ContainerDied","Data":"ea5977599a6e28a0b3824fad9e4edf89228d43b8b19addd0e52f39e25c8dcb6a"}
Jul 13 19:02:13.512332 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.512068    2581 scope.go:117] "RemoveContainer" containerID="4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5"
Jul 13 19:02:13.520986 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.520965    2581 scope.go:117] "RemoveContainer" containerID="4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5"
Jul 13 19:02:13.521282 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:02:13.521259    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5\": container with ID starting with 4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5 not found: ID does not exist" containerID="4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5"
Jul 13 19:02:13.521342 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.521291    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5"} err="failed to get container status \"4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5\": rpc error: code = NotFound desc = could not find container \"4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5\": container with ID starting with 4a8d47b1e6c0735976b20d1f843fed890ae4a67c370c4402ebecf4f811e137e5 not found: ID does not exist"
Jul 13 19:02:13.532609 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.532586    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"]
Jul 13 19:02:13.536218 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.536191    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-4ph8z"]
Jul 13 19:02:13.603851 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:13.603817    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df03b34-c12d-4743-bfa7-c49f037b66b2" path="/var/lib/kubelet/pods/7df03b34-c12d-4743-bfa7-c49f037b66b2/volumes"
Jul 13 19:02:18.481073 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:18.481034    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:18.582058 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:18.582006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:28.313617 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.313582    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-w9wkj"]
Jul 13 19:02:28.314008 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.313880    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df03b34-c12d-4743-bfa7-c49f037b66b2" containerName="limitador"
Jul 13 19:02:28.314008 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.313892    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df03b34-c12d-4743-bfa7-c49f037b66b2" containerName="limitador"
Jul 13 19:02:28.314008 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.313948    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df03b34-c12d-4743-bfa7-c49f037b66b2" containerName="limitador"
Jul 13 19:02:28.317906 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.317890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.320344 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.320319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 13 19:02:28.320488 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.320346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 13 19:02:28.320488 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.320360    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-4cvbg\""
Jul 13 19:02:28.320488 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.320381    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 13 19:02:28.327169 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.327149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-w9wkj"]
Jul 13 19:02:28.351766 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.351733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc4157cb-5aec-4bc3-b7bf-d33f146aefa7-cert\") pod \"kserve-controller-manager-6b69b6d547-w9wkj\" (UID: \"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7\") " pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.351947 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.351799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwqz\" (UniqueName: \"kubernetes.io/projected/bc4157cb-5aec-4bc3-b7bf-d33f146aefa7-kube-api-access-gnwqz\") pod \"kserve-controller-manager-6b69b6d547-w9wkj\" (UID: \"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7\") " pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.452268 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.452232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc4157cb-5aec-4bc3-b7bf-d33f146aefa7-cert\") pod \"kserve-controller-manager-6b69b6d547-w9wkj\" (UID: \"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7\") " pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.452457 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.452279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwqz\" (UniqueName: \"kubernetes.io/projected/bc4157cb-5aec-4bc3-b7bf-d33f146aefa7-kube-api-access-gnwqz\") pod \"kserve-controller-manager-6b69b6d547-w9wkj\" (UID: \"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7\") " pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.454729 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.454702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc4157cb-5aec-4bc3-b7bf-d33f146aefa7-cert\") pod \"kserve-controller-manager-6b69b6d547-w9wkj\" (UID: \"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7\") " pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.460647 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.460622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwqz\" (UniqueName: \"kubernetes.io/projected/bc4157cb-5aec-4bc3-b7bf-d33f146aefa7-kube-api-access-gnwqz\") pod \"kserve-controller-manager-6b69b6d547-w9wkj\" (UID: \"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7\") " pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.628812 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.628704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:28.764852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.764826    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-w9wkj"]
Jul 13 19:02:28.767018 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:02:28.766991    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4157cb_5aec_4bc3_b7bf_d33f146aefa7.slice/crio-ed29999539485d8b41f8c7a9c18f964fff4acb9f4f69e65fe18d7065a91f9ce7 WatchSource:0}: Error finding container ed29999539485d8b41f8c7a9c18f964fff4acb9f4f69e65fe18d7065a91f9ce7: Status 404 returned error can't find the container with id ed29999539485d8b41f8c7a9c18f964fff4acb9f4f69e65fe18d7065a91f9ce7
Jul 13 19:02:28.768872 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:28.768854    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 19:02:29.596186 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:29.596147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj" event={"ID":"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7","Type":"ContainerStarted","Data":"ed29999539485d8b41f8c7a9c18f964fff4acb9f4f69e65fe18d7065a91f9ce7"}
Jul 13 19:02:31.605905 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:31.605862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj" event={"ID":"bc4157cb-5aec-4bc3-b7bf-d33f146aefa7","Type":"ContainerStarted","Data":"c3124eb1807dd1264201fd2c356393a2190cef36eefaf14724013ebaf18699e4"}
Jul 13 19:02:31.606258 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:31.605923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:02:31.624196 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:31.624149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj" podStartSLOduration=0.97429741 podStartE2EDuration="3.624133229s" totalImagesPullingTime="2.649835819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:02:31.622515445 +0000 UTC m=+788.522740836" watchObservedRunningTime="2026-07-13 19:02:31.624133229 +0000 UTC m=+788.524358621"
Jul 13 19:02:51.033333 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:51.033240    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:51.096187 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:02:51.096150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:02.616414 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:02.616383    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b69b6d547-w9wkj"
Jul 13 19:03:03.620743 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.620706    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:03.693010 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.692977    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-c4ptz"]
Jul 13 19:03:03.697632 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.697615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:03.702911 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.702886    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-f2kqv\""
Jul 13 19:03:03.703043 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.702884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 13 19:03:03.711130 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.711108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-c4ptz"]
Jul 13 19:03:03.723410 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.723374    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-gpt9r"]
Jul 13 19:03:03.726538 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.726524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:03.730429 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.730409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-mbdq5\""
Jul 13 19:03:03.730534 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.730453    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 13 19:03:03.758991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.758963    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-gpt9r"]
Jul 13 19:03:03.794286 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.794255    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:03.827322 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.827288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4n77\" (UniqueName: \"kubernetes.io/projected/6f512d55-c310-430a-b87f-cce0542b2ef4-kube-api-access-k4n77\") pod \"odh-model-controller-84b54b7f97-gpt9r\" (UID: \"6f512d55-c310-430a-b87f-cce0542b2ef4\") " pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:03.827322 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.827330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cd837e-531b-4956-b9b2-7ff100b732db-tls-certs\") pod \"model-serving-api-fd65d7d6-c4ptz\" (UID: \"41cd837e-531b-4956-b9b2-7ff100b732db\") " pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:03.827526 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.827361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f512d55-c310-430a-b87f-cce0542b2ef4-cert\") pod \"odh-model-controller-84b54b7f97-gpt9r\" (UID: \"6f512d55-c310-430a-b87f-cce0542b2ef4\") " pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:03.827526 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.827423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djj5b\" (UniqueName: \"kubernetes.io/projected/41cd837e-531b-4956-b9b2-7ff100b732db-kube-api-access-djj5b\") pod \"model-serving-api-fd65d7d6-c4ptz\" (UID: \"41cd837e-531b-4956-b9b2-7ff100b732db\") " pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:03.928737 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.928662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4n77\" (UniqueName: \"kubernetes.io/projected/6f512d55-c310-430a-b87f-cce0542b2ef4-kube-api-access-k4n77\") pod \"odh-model-controller-84b54b7f97-gpt9r\" (UID: \"6f512d55-c310-430a-b87f-cce0542b2ef4\") " pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:03.928737 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.928720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cd837e-531b-4956-b9b2-7ff100b732db-tls-certs\") pod \"model-serving-api-fd65d7d6-c4ptz\" (UID: \"41cd837e-531b-4956-b9b2-7ff100b732db\") " pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:03.928920 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.928771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f512d55-c310-430a-b87f-cce0542b2ef4-cert\") pod \"odh-model-controller-84b54b7f97-gpt9r\" (UID: \"6f512d55-c310-430a-b87f-cce0542b2ef4\") " pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:03.928920 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.928829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djj5b\" (UniqueName: \"kubernetes.io/projected/41cd837e-531b-4956-b9b2-7ff100b732db-kube-api-access-djj5b\") pod \"model-serving-api-fd65d7d6-c4ptz\" (UID: \"41cd837e-531b-4956-b9b2-7ff100b732db\") " pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:03.928920 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:03:03.928888    2581 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 13 19:03:03.929018 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:03:03.928933    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 13 19:03:03.929018 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:03:03.928945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cd837e-531b-4956-b9b2-7ff100b732db-tls-certs podName:41cd837e-531b-4956-b9b2-7ff100b732db nodeName:}" failed. No retries permitted until 2026-07-13 19:03:04.428929927 +0000 UTC m=+821.329155297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/41cd837e-531b-4956-b9b2-7ff100b732db-tls-certs") pod "model-serving-api-fd65d7d6-c4ptz" (UID: "41cd837e-531b-4956-b9b2-7ff100b732db") : secret "model-serving-api-tls" not found
Jul 13 19:03:03.929018 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:03:03.928976    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f512d55-c310-430a-b87f-cce0542b2ef4-cert podName:6f512d55-c310-430a-b87f-cce0542b2ef4 nodeName:}" failed. No retries permitted until 2026-07-13 19:03:04.428964058 +0000 UTC m=+821.329189432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6f512d55-c310-430a-b87f-cce0542b2ef4-cert") pod "odh-model-controller-84b54b7f97-gpt9r" (UID: "6f512d55-c310-430a-b87f-cce0542b2ef4") : secret "odh-model-controller-webhook-cert" not found
Jul 13 19:03:03.937867 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.937838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4n77\" (UniqueName: \"kubernetes.io/projected/6f512d55-c310-430a-b87f-cce0542b2ef4-kube-api-access-k4n77\") pod \"odh-model-controller-84b54b7f97-gpt9r\" (UID: \"6f512d55-c310-430a-b87f-cce0542b2ef4\") " pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:03.937982 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:03.937922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djj5b\" (UniqueName: \"kubernetes.io/projected/41cd837e-531b-4956-b9b2-7ff100b732db-kube-api-access-djj5b\") pod \"model-serving-api-fd65d7d6-c4ptz\" (UID: \"41cd837e-531b-4956-b9b2-7ff100b732db\") " pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:04.102633 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.102603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:04.205178 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.205147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:04.429191 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.429160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:04.433766 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.433744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cd837e-531b-4956-b9b2-7ff100b732db-tls-certs\") pod \"model-serving-api-fd65d7d6-c4ptz\" (UID: \"41cd837e-531b-4956-b9b2-7ff100b732db\") " pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:04.433910 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.433806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f512d55-c310-430a-b87f-cce0542b2ef4-cert\") pod \"odh-model-controller-84b54b7f97-gpt9r\" (UID: \"6f512d55-c310-430a-b87f-cce0542b2ef4\") " pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:04.436223 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.436203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f512d55-c310-430a-b87f-cce0542b2ef4-cert\") pod \"odh-model-controller-84b54b7f97-gpt9r\" (UID: \"6f512d55-c310-430a-b87f-cce0542b2ef4\") " pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:04.436311 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.436290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cd837e-531b-4956-b9b2-7ff100b732db-tls-certs\") pod \"model-serving-api-fd65d7d6-c4ptz\" (UID: \"41cd837e-531b-4956-b9b2-7ff100b732db\") " pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:04.490359 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.490275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:04.607669 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.607633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:04.636777 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.636746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:04.756673 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.756645    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-c4ptz"]
Jul 13 19:03:04.759113 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:03:04.759087    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cd837e_531b_4956_b9b2_7ff100b732db.slice/crio-c205ca176ab17802950dd925ff8ce3893892091192f716a25b3fdf61257c1354 WatchSource:0}: Error finding container c205ca176ab17802950dd925ff8ce3893892091192f716a25b3fdf61257c1354: Status 404 returned error can't find the container with id c205ca176ab17802950dd925ff8ce3893892091192f716a25b3fdf61257c1354
Jul 13 19:03:04.779385 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:04.779362    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-gpt9r"]
Jul 13 19:03:04.781270 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:03:04.781246    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f512d55_c310_430a_b87f_cce0542b2ef4.slice/crio-d66598d225e48e6b983613ab8c1da8dbc296ef9e524193606e7fd3a2214e08d9 WatchSource:0}: Error finding container d66598d225e48e6b983613ab8c1da8dbc296ef9e524193606e7fd3a2214e08d9: Status 404 returned error can't find the container with id d66598d225e48e6b983613ab8c1da8dbc296ef9e524193606e7fd3a2214e08d9
Jul 13 19:03:05.767961 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:05.767926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-gpt9r" event={"ID":"6f512d55-c310-430a-b87f-cce0542b2ef4","Type":"ContainerStarted","Data":"d66598d225e48e6b983613ab8c1da8dbc296ef9e524193606e7fd3a2214e08d9"}
Jul 13 19:03:05.769424 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:05.769390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-c4ptz" event={"ID":"41cd837e-531b-4956-b9b2-7ff100b732db","Type":"ContainerStarted","Data":"c205ca176ab17802950dd925ff8ce3893892091192f716a25b3fdf61257c1354"}
Jul 13 19:03:08.787493 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:08.787461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-gpt9r" event={"ID":"6f512d55-c310-430a-b87f-cce0542b2ef4","Type":"ContainerStarted","Data":"9d65002a518d73e20c3dbe2a18659d748311fc8663cd64c67a446c76ce6bb622"}
Jul 13 19:03:08.788023 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:08.787552    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:08.789005 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:08.788982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-c4ptz" event={"ID":"41cd837e-531b-4956-b9b2-7ff100b732db","Type":"ContainerStarted","Data":"933cee1d775c0369c667e0dc6512ee459e639d09270267333c8ab6af71f2b309"}
Jul 13 19:03:08.789135 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:08.789078    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:03:08.805846 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:08.805750    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-gpt9r" podStartSLOduration=2.614390997 podStartE2EDuration="5.805736764s" totalImagesPullingTime="3.191345767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:03:08.804348255 +0000 UTC m=+825.704573647" watchObservedRunningTime="2026-07-13 19:03:08.805736764 +0000 UTC m=+825.705962155"
Jul 13 19:03:08.823377 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:08.823334    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-c4ptz" podStartSLOduration=2.657079289 podStartE2EDuration="5.823320826s" totalImagesPullingTime="3.166241537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:03:08.82163618 +0000 UTC m=+825.721861580" watchObservedRunningTime="2026-07-13 19:03:08.823320826 +0000 UTC m=+825.723546217"
Jul 13 19:03:19.796341 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:19.796311    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-gpt9r"
Jul 13 19:03:19.799127 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:03:19.799105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-c4ptz"
Jul 13 19:04:23.571947 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:23.571875    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:04:23.571947 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:23.571887    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:04:41.456666 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:41.456634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:42.604382 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:42.604348    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:42.691267 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:42.691237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:55.486508 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.486474    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"]
Jul 13 19:04:55.489847 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.489828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.492246 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.492224    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:04:55.492701 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.492680    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:04:55.492867 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.492697    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zh96m\""
Jul 13 19:04:55.492981 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.492709    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 13 19:04:55.508781 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.508740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"]
Jul 13 19:04:55.522888 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.522843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:55.605768 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkp64\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-kube-api-access-pkp64\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.605990 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.605974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.620355 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.620327    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:55.694125 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.694087    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"]
Jul 13 19:04:55.697432 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.697406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.707102 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707255 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707255 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707255 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707423 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkp64\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-kube-api-access-pkp64\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707482 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707482 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707585 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707585 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.707925 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.708084 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.707919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.708154 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.708127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.708654 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.708601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.709257 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.709128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.710577 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.710554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.710966 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.710941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.712218 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.712194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"]
Jul 13 19:04:55.721496 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.721013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkp64\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-kube-api-access-pkp64\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.721496 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.721438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-kkqwb\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.803013 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.802931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:04:55.809672 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.809822 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.809822 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.809822 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.809822 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58mj\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-kube-api-access-t58mj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.810048 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.810048 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.810048 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.810048 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.809983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8342c50f-19eb-459d-8e6e-bb5ea45858de-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8342c50f-19eb-459d-8e6e-bb5ea45858de-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58mj\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-kube-api-access-t58mj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.911916 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.911844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.912354 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.912084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.912701 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.912673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.914049 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.914010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.914605 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.914578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8342c50f-19eb-459d-8e6e-bb5ea45858de-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.914872 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.914851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.915294 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.915270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.920059 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.920031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.921058 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.921035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58mj\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-kube-api-access-t58mj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-lfprg\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:55.949823 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:55.949781    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"]
Jul 13 19:04:55.951778 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:04:55.951751    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde72d2c2_1ff5_4f8f_a908_a5b0e57b2bda.slice/crio-ca3763fe211932b2acb9f7b437b078ba3377915549d0de1cd6b397c5a01b56df WatchSource:0}: Error finding container ca3763fe211932b2acb9f7b437b078ba3377915549d0de1cd6b397c5a01b56df: Status 404 returned error can't find the container with id ca3763fe211932b2acb9f7b437b078ba3377915549d0de1cd6b397c5a01b56df
Jul 13 19:04:56.010332 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:56.010287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:04:56.154385 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:56.154357    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"]
Jul 13 19:04:56.154927 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:04:56.154900    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8342c50f_19eb_459d_8e6e_bb5ea45858de.slice/crio-1706c0be2b1c3ebe46d27b72c4bb7f0744718586a6c9c3e56d78610ef9069d6b WatchSource:0}: Error finding container 1706c0be2b1c3ebe46d27b72c4bb7f0744718586a6c9c3e56d78610ef9069d6b: Status 404 returned error can't find the container with id 1706c0be2b1c3ebe46d27b72c4bb7f0744718586a6c9c3e56d78610ef9069d6b
Jul 13 19:04:56.291108 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:56.291071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb" event={"ID":"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda","Type":"ContainerStarted","Data":"ca3763fe211932b2acb9f7b437b078ba3377915549d0de1cd6b397c5a01b56df"}
Jul 13 19:04:56.292283 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:56.292259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg" event={"ID":"8342c50f-19eb-459d-8e6e-bb5ea45858de","Type":"ContainerStarted","Data":"1706c0be2b1c3ebe46d27b72c4bb7f0744718586a6c9c3e56d78610ef9069d6b"}
Jul 13 19:04:59.296623 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:59.296576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:59.660855 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:59.660817    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:04:59.660992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:59.660887    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:04:59.660992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:59.660921    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:04:59.670940 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:59.670911    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:04:59.671028 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:59.670969    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:04:59.671028 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:04:59.671015    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:00.316821 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:00.316770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb" event={"ID":"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda","Type":"ContainerStarted","Data":"a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37"}
Jul 13 19:05:00.318489 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:00.318459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg" event={"ID":"8342c50f-19eb-459d-8e6e-bb5ea45858de","Type":"ContainerStarted","Data":"5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46"}
Jul 13 19:05:00.340921 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:00.340857    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb" podStartSLOduration=1.634119793 podStartE2EDuration="5.340838454s" totalImagesPullingTime="3.706718661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:00.339369089 +0000 UTC m=+937.239594481" watchObservedRunningTime="2026-07-13 19:05:00.340838454 +0000 UTC m=+937.241063848"
Jul 13 19:05:00.360282 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:00.360227    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg" podStartSLOduration=1.846477917 podStartE2EDuration="5.360214232s" totalImagesPullingTime="3.513736315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:00.358534041 +0000 UTC m=+937.258759432" watchObservedRunningTime="2026-07-13 19:05:00.360214232 +0000 UTC m=+937.260439625"
Jul 13 19:05:00.803328 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:00.803288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:05:00.808557 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:00.808530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:05:01.010948 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:01.010905    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:05:01.016303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:01.016279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:05:01.323714 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:01.323675    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:05:01.324181 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:01.323735    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:05:01.324565 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:01.324544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:05:01.324689 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:01.324671    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:05:01.412981 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:01.412944    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"]
Jul 13 19:05:02.743668 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:02.743632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:03.333677 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:03.333633    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb" podUID="de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" containerName="istio-proxy" containerID="cri-o://a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37" gracePeriod=30
Jul 13 19:05:06.383072 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:06.383029    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"]
Jul 13 19:05:06.383543 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:06.383260    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg" podUID="8342c50f-19eb-459d-8e6e-bb5ea45858de" containerName="istio-proxy" containerID="cri-o://5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46" gracePeriod=30
Jul 13 19:05:08.621799 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.621757    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:05:08.732527 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istiod-ca-cert\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732709 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-credential-socket\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732709 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-socket\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732709 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-certs\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732907 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-token\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732907 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-data\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732907 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-envoy\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732907 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkp64\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-kube-api-access-pkp64\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.732907 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732891    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-podinfo\") pod \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\" (UID: \"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda\") "
Jul 13 19:05:08.733126 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:08.733126 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732929    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:05:08.733126 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.732947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:08.733126 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.733091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:08.733303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.733174    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istiod-ca-cert\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.733303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.733194    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-credential-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.733303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.733213    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.733303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.733227    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-workload-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.733503 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.733480    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-data" (OuterVolumeSpecName: "istio-data") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:08.735501 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.735467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 19:05:08.735628 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.735573    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-token" (OuterVolumeSpecName: "istio-token") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:05:08.735701 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.735678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-kube-api-access-pkp64" (OuterVolumeSpecName: "kube-api-access-pkp64") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "kube-api-access-pkp64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:05:08.735757 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.735709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" (UID: "de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:08.834314 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.834277    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-token\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.834314 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.834305    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-data\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.834314 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.834315    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-envoy\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.834535 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.834325    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkp64\" (UniqueName: \"kubernetes.io/projected/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-kube-api-access-pkp64\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:08.834535 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:08.834334    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda-istio-podinfo\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:09.369622 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.369587    2581 generic.go:356] "Generic (PLEG): container finished" podID="de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" containerID="a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37" exitCode=0
Jul 13 19:05:09.369811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.369637    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"
Jul 13 19:05:09.369811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.369670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb" event={"ID":"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda","Type":"ContainerDied","Data":"a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37"}
Jul 13 19:05:09.369811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.369723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb" event={"ID":"de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda","Type":"ContainerDied","Data":"ca3763fe211932b2acb9f7b437b078ba3377915549d0de1cd6b397c5a01b56df"}
Jul 13 19:05:09.369811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.369746    2581 scope.go:117] "RemoveContainer" containerID="a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37"
Jul 13 19:05:09.384469 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.384451    2581 scope.go:117] "RemoveContainer" containerID="a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37"
Jul 13 19:05:09.384779 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:05:09.384756    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37\": container with ID starting with a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37 not found: ID does not exist" containerID="a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37"
Jul 13 19:05:09.384875 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.384813    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37"} err="failed to get container status \"a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37\": rpc error: code = NotFound desc = could not find container \"a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37\": container with ID starting with a4932efca86168cc02c3612129ac52f8dd39259280f6f48fbed9bbc2320bfd37 not found: ID does not exist"
Jul 13 19:05:09.401009 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.400977    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"]
Jul 13 19:05:09.403953 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.403924    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb"]
Jul 13 19:05:09.605670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:09.605638    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" path="/var/lib/kubelet/pods/de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda/volumes"
Jul 13 19:05:12.738120 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.738061    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:05:12.869674 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-podinfo\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869674 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8342c50f-19eb-459d-8e6e-bb5ea45858de-istiod-ca-cert\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869674 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t58mj\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-kube-api-access-t58mj\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869684    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-certs\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869707    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-socket\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-envoy\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-token\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-data\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.869992 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.869884    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-credential-socket\") pod \"8342c50f-19eb-459d-8e6e-bb5ea45858de\" (UID: \"8342c50f-19eb-459d-8e6e-bb5ea45858de\") "
Jul 13 19:05:12.870279 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.870150    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8342c50f-19eb-459d-8e6e-bb5ea45858de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:05:12.870279 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.870195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:12.870279 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.870252    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-data" (OuterVolumeSpecName: "istio-data") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:12.870595 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.870273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:12.870595 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.870287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:12.872052 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.872019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-token" (OuterVolumeSpecName: "istio-token") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:05:12.872173 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.872099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 19:05:12.872236 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.872170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-kube-api-access-t58mj" (OuterVolumeSpecName: "kube-api-access-t58mj") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "kube-api-access-t58mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:05:12.872336 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.872315    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8342c50f-19eb-459d-8e6e-bb5ea45858de" (UID: "8342c50f-19eb-459d-8e6e-bb5ea45858de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:12.971195 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971146    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-podinfo\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971195 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971188    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8342c50f-19eb-459d-8e6e-bb5ea45858de-istiod-ca-cert\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971195 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971201    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t58mj\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-kube-api-access-t58mj\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971456 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971214    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971456 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971226    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-workload-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971456 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971237    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-envoy\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971456 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971248    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-token\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971456 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971258    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-istio-data\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:12.971456 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:12.971268    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8342c50f-19eb-459d-8e6e-bb5ea45858de-credential-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:05:13.393187 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.393150    2581 generic.go:356] "Generic (PLEG): container finished" podID="8342c50f-19eb-459d-8e6e-bb5ea45858de" containerID="5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46" exitCode=0
Jul 13 19:05:13.393336 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.393204    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"
Jul 13 19:05:13.393336 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.393239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg" event={"ID":"8342c50f-19eb-459d-8e6e-bb5ea45858de","Type":"ContainerDied","Data":"5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46"}
Jul 13 19:05:13.393336 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.393313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg" event={"ID":"8342c50f-19eb-459d-8e6e-bb5ea45858de","Type":"ContainerDied","Data":"1706c0be2b1c3ebe46d27b72c4bb7f0744718586a6c9c3e56d78610ef9069d6b"}
Jul 13 19:05:13.393472 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.393336    2581 scope.go:117] "RemoveContainer" containerID="5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46"
Jul 13 19:05:13.402533 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.402519    2581 scope.go:117] "RemoveContainer" containerID="5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46"
Jul 13 19:05:13.402778 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:05:13.402757    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46\": container with ID starting with 5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46 not found: ID does not exist" containerID="5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46"
Jul 13 19:05:13.402865 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.402805    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46"} err="failed to get container status \"5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46\": rpc error: code = NotFound desc = could not find container \"5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46\": container with ID starting with 5af9b2c1ef6989ed92c3a39c2088bce8b34409560c472ddcc7a0353878139b46 not found: ID does not exist"
Jul 13 19:05:13.417379 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.417356    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"]
Jul 13 19:05:13.421548 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.421514    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg"]
Jul 13 19:05:13.603805 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.603751    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8342c50f-19eb-459d-8e6e-bb5ea45858de" path="/var/lib/kubelet/pods/8342c50f-19eb-459d-8e6e-bb5ea45858de/volumes"
Jul 13 19:05:13.979994 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.979964    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"]
Jul 13 19:05:13.980460 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.980403    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8342c50f-19eb-459d-8e6e-bb5ea45858de" containerName="istio-proxy"
Jul 13 19:05:13.980460 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.980420    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8342c50f-19eb-459d-8e6e-bb5ea45858de" containerName="istio-proxy"
Jul 13 19:05:13.980460 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.980453    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" containerName="istio-proxy"
Jul 13 19:05:13.980460 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.980462    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" containerName="istio-proxy"
Jul 13 19:05:13.980668 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.980538    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8342c50f-19eb-459d-8e6e-bb5ea45858de" containerName="istio-proxy"
Jul 13 19:05:13.980668 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.980551    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de72d2c2-1ff5-4f8f-a908-a5b0e57b2bda" containerName="istio-proxy"
Jul 13 19:05:13.984972 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.984948    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:13.987692 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.987671    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:05:13.988406 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.988387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:05:13.988499 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.988436    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:05:13.988499 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.988453    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:05:13.988601 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:13.988506    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 13 19:05:14.000473 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.000445    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"]
Jul 13 19:05:14.029935 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.029899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:14.078704 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.078666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-home\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.078894 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.078711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tls-certs\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.078894 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.078737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-model-cache\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.078894 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.078771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.078894 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.078816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885gc\" (UniqueName: \"kubernetes.io/projected/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kube-api-access-885gc\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.078894 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.078848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.078894 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.078893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-dshm\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.179939 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.179901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-home\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.179939 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.179946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tls-certs\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180158 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.179971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-model-cache\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180158 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180158 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885gc\" (UniqueName: \"kubernetes.io/projected/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kube-api-access-885gc\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180158 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180158 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-dshm\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180397 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-model-cache\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180459 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180515 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-home\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.180571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.180549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.182452 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.182433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-dshm\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.182534 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.182501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tls-certs\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.189206 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.189172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-885gc\" (UniqueName: \"kubernetes.io/projected/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kube-api-access-885gc\") pod \"precise-prefix-cache-test-kserve-6ffc955574-v494f\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.296217 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.296127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:14.383433 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.379814    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"]
Jul 13 19:05:14.386716 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.386643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.390336 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.389454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-jvzmw\""
Jul 13 19:05:14.398335 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.397155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"]
Jul 13 19:05:14.403584 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.403539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:14.451565 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.451541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"]
Jul 13 19:05:14.453425 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:05:14.453399    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7b8723_9dc9_4f4f_aa9a_4e9f4098d2df.slice/crio-e6512eb4b71e848252425e47b2b622e336e39ea403fdf429bd1ea8a45fc2b569 WatchSource:0}: Error finding container e6512eb4b71e848252425e47b2b622e336e39ea403fdf429bd1ea8a45fc2b569: Status 404 returned error can't find the container with id e6512eb4b71e848252425e47b2b622e336e39ea403fdf429bd1ea8a45fc2b569
Jul 13 19:05:14.485180 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.485149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.485344 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.485194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8461ba0-2951-4344-a855-6a37faceee62-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.485344 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.485219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.485344 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.485303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.485481 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.485350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shccl\" (UniqueName: \"kubernetes.io/projected/b8461ba0-2951-4344-a855-6a37faceee62-kube-api-access-shccl\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.485481 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.485415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.586665 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.586563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.586665 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.586617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.586995 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.586698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shccl\" (UniqueName: \"kubernetes.io/projected/b8461ba0-2951-4344-a855-6a37faceee62-kube-api-access-shccl\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.586995 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.586757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.586995 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.586864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.586995 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.586906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8461ba0-2951-4344-a855-6a37faceee62-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.587204 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.587097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.587204 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.587099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.587204 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.587147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.587297 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.587274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.589588 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.589560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8461ba0-2951-4344-a855-6a37faceee62-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.595936 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.595904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shccl\" (UniqueName: \"kubernetes.io/projected/b8461ba0-2951-4344-a855-6a37faceee62-kube-api-access-shccl\") pod \"precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.702527 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.702490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:14.851587 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:14.851561    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"]
Jul 13 19:05:14.853186 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:05:14.853159    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8461ba0_2951_4344_a855_6a37faceee62.slice/crio-b40961bb8054c330fbbde34d2808be316d991c827c5c3cc48c8f822135e9eae6 WatchSource:0}: Error finding container b40961bb8054c330fbbde34d2808be316d991c827c5c3cc48c8f822135e9eae6: Status 404 returned error can't find the container with id b40961bb8054c330fbbde34d2808be316d991c827c5c3cc48c8f822135e9eae6
Jul 13 19:05:15.411076 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:15.411041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerStarted","Data":"b40961bb8054c330fbbde34d2808be316d991c827c5c3cc48c8f822135e9eae6"}
Jul 13 19:05:15.412233 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:15.412207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" event={"ID":"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df","Type":"ContainerStarted","Data":"e6512eb4b71e848252425e47b2b622e336e39ea403fdf429bd1ea8a45fc2b569"}
Jul 13 19:05:16.627148 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:16.627110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:20.444670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:20.444630    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8461ba0-2951-4344-a855-6a37faceee62" containerID="9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd" exitCode=0
Jul 13 19:05:20.445120 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:20.444724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerDied","Data":"9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd"}
Jul 13 19:05:20.446308 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:20.446286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" event={"ID":"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df","Type":"ContainerStarted","Data":"c750b83aedaf25630411eff2a3b9c4315ba531743f1544ebe66b8c57a2baace6"}
Jul 13 19:05:20.533297 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:20.533262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:22.464096 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:22.464057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerStarted","Data":"94170164c058710790b40f530c4eb378374c89952d11b6082e8335ae3b485e2f"}
Jul 13 19:05:23.472593 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:23.472553    2581 generic.go:356] "Generic (PLEG): container finished" podID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerID="c750b83aedaf25630411eff2a3b9c4315ba531743f1544ebe66b8c57a2baace6" exitCode=0
Jul 13 19:05:23.473089 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:23.472620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" event={"ID":"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df","Type":"ContainerDied","Data":"c750b83aedaf25630411eff2a3b9c4315ba531743f1544ebe66b8c57a2baace6"}
Jul 13 19:05:25.488545 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:25.488507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" event={"ID":"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df","Type":"ContainerStarted","Data":"20c76622ad13696fe30daf6d277a308c1568df21eda3c930ff493ddfca135ad9"}
Jul 13 19:05:25.513581 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:25.513530    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" podStartSLOduration=2.470003917 podStartE2EDuration="12.51351131s" totalImagesPullingTime="6.953304352s" totalInitContainerRuntime="3.090203041s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:05:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:25.510927535 +0000 UTC m=+962.411152928" watchObservedRunningTime="2026-07-13 19:05:25.51351131 +0000 UTC m=+962.413736712"
Jul 13 19:05:26.296588 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:26.296552    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:32.533839 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:32.533747    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw_b8461ba0-2951-4344-a855-6a37faceee62/main/0.log"
Jul 13 19:05:32.534278 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:32.534136    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8461ba0-2951-4344-a855-6a37faceee62" containerID="94170164c058710790b40f530c4eb378374c89952d11b6082e8335ae3b485e2f" exitCode=1
Jul 13 19:05:32.534278 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:32.534211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerDied","Data":"94170164c058710790b40f530c4eb378374c89952d11b6082e8335ae3b485e2f"}
Jul 13 19:05:34.297142 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:34.297100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:34.313020 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:34.312989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:34.563311 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:34.563232    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:05:48.726537 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:48.726443    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:50.493822 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:50.493770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:52.653749 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:52.653667    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw_b8461ba0-2951-4344-a855-6a37faceee62/main/0.log"
Jul 13 19:05:52.654187 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:52.654165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerStarted","Data":"b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d"}
Jul 13 19:05:52.654563 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:52.654545    2581 scope.go:117] "RemoveContainer" containerID="94170164c058710790b40f530c4eb378374c89952d11b6082e8335ae3b485e2f"
Jul 13 19:05:53.660969 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:53.660940    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw_b8461ba0-2951-4344-a855-6a37faceee62/main/0.log"
Jul 13 19:05:53.661477 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:53.661453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerStarted","Data":"a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120"}
Jul 13 19:05:53.661550 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:53.661538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:53.687116 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:53.687047    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" podStartSLOduration=2.380818782 podStartE2EDuration="39.687005256s" totalImagesPullingTime="36.592852223s" totalInitContainerRuntime="713.334251ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:05:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:53.685422355 +0000 UTC m=+990.585647746" watchObservedRunningTime="2026-07-13 19:05:53.687005256 +0000 UTC m=+990.587230649"
Jul 13 19:05:54.703113 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:54.703075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:05:54.703483 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:05:54.703161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:06:04.704373 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:06:04.704329    2581 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.34:9003", ServerName: "10.132.0.34:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.34:9003: connect: connection refused"
Jul 13 19:06:04.705643 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:04.705621    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:06:04.706947 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:04.706924    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:06:05.703343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:05.703306    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.34:9003\" within 1s: context deadline exceeded"
Jul 13 19:06:07.738579 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:07.738542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:07.955452 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:07.955400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:18.535056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:18.535009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:24.717967 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:24.717937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:06:26.690139 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.690103    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"]
Jul 13 19:06:26.701036 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.701006    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"]
Jul 13 19:06:26.701298 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.701271    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="tokenizer" containerID="cri-o://b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d" gracePeriod=60
Jul 13 19:06:26.710962 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:06:26.710927    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:06:26.710962 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:06:26.710962 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:06:26.710962 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120" cmd=["/bin/sleep","15"]
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.710984    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" message="" err=<
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:06:26.710997    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main" containerID="cri-o://a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120"
Jul 13 19:06:26.711215 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.711043    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main" containerID="cri-o://a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120" gracePeriod=60
Jul 13 19:06:26.746627 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.746586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:26.800655 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.800625    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:26.829707 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.829684    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw_b8461ba0-2951-4344-a855-6a37faceee62/main/0.log"
Jul 13 19:06:26.830080 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.830057    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8461ba0-2951-4344-a855-6a37faceee62" containerID="a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120" exitCode=0
Jul 13 19:06:26.830184 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.830099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerDied","Data":"a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120"}
Jul 13 19:06:26.830184 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:26.830154    2581 scope.go:117] "RemoveContainer" containerID="94170164c058710790b40f530c4eb378374c89952d11b6082e8335ae3b485e2f"
Jul 13 19:06:28.256028 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.256000    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:06:28.322155 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-uds\") pod \"b8461ba0-2951-4344-a855-6a37faceee62\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") "
Jul 13 19:06:28.322155 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-kserve-provision-location\") pod \"b8461ba0-2951-4344-a855-6a37faceee62\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") "
Jul 13 19:06:28.322592 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8461ba0-2951-4344-a855-6a37faceee62-tls-certs\") pod \"b8461ba0-2951-4344-a855-6a37faceee62\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") "
Jul 13 19:06:28.322592 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shccl\" (UniqueName: \"kubernetes.io/projected/b8461ba0-2951-4344-a855-6a37faceee62-kube-api-access-shccl\") pod \"b8461ba0-2951-4344-a855-6a37faceee62\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") "
Jul 13 19:06:28.322592 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322241    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-tmp\") pod \"b8461ba0-2951-4344-a855-6a37faceee62\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") "
Jul 13 19:06:28.322592 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-cache\") pod \"b8461ba0-2951-4344-a855-6a37faceee62\" (UID: \"b8461ba0-2951-4344-a855-6a37faceee62\") "
Jul 13 19:06:28.322592 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322389    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b8461ba0-2951-4344-a855-6a37faceee62" (UID: "b8461ba0-2951-4344-a855-6a37faceee62"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:28.322592 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322550    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:28.322983 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b8461ba0-2951-4344-a855-6a37faceee62" (UID: "b8461ba0-2951-4344-a855-6a37faceee62"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:28.322983 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322723    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b8461ba0-2951-4344-a855-6a37faceee62" (UID: "b8461ba0-2951-4344-a855-6a37faceee62"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:28.322983 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.322950    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8461ba0-2951-4344-a855-6a37faceee62" (UID: "b8461ba0-2951-4344-a855-6a37faceee62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:28.324474 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.324448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8461ba0-2951-4344-a855-6a37faceee62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8461ba0-2951-4344-a855-6a37faceee62" (UID: "b8461ba0-2951-4344-a855-6a37faceee62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:06:28.324564 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.324539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8461ba0-2951-4344-a855-6a37faceee62-kube-api-access-shccl" (OuterVolumeSpecName: "kube-api-access-shccl") pod "b8461ba0-2951-4344-a855-6a37faceee62" (UID: "b8461ba0-2951-4344-a855-6a37faceee62"). InnerVolumeSpecName "kube-api-access-shccl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:06:28.423863 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.423825    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8461ba0-2951-4344-a855-6a37faceee62-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:28.423863 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.423858    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shccl\" (UniqueName: \"kubernetes.io/projected/b8461ba0-2951-4344-a855-6a37faceee62-kube-api-access-shccl\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:28.423863 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.423869    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-tmp\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:28.424122 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.423879    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-tokenizer-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:28.424122 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.423888    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8461ba0-2951-4344-a855-6a37faceee62-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:28.844114 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.844075    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8461ba0-2951-4344-a855-6a37faceee62" containerID="b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d" exitCode=0
Jul 13 19:06:28.844295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.844117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"
Jul 13 19:06:28.844295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.844125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerDied","Data":"b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d"}
Jul 13 19:06:28.844295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.844170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw" event={"ID":"b8461ba0-2951-4344-a855-6a37faceee62","Type":"ContainerDied","Data":"b40961bb8054c330fbbde34d2808be316d991c827c5c3cc48c8f822135e9eae6"}
Jul 13 19:06:28.844295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.844193    2581 scope.go:117] "RemoveContainer" containerID="a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120"
Jul 13 19:06:28.853100 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.853082    2581 scope.go:117] "RemoveContainer" containerID="b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d"
Jul 13 19:06:28.860628 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.860607    2581 scope.go:117] "RemoveContainer" containerID="9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd"
Jul 13 19:06:28.866395 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.866373    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"]
Jul 13 19:06:28.868879 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.868860    2581 scope.go:117] "RemoveContainer" containerID="a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120"
Jul 13 19:06:28.869135 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:06:28.869119    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120\": container with ID starting with a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120 not found: ID does not exist" containerID="a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120"
Jul 13 19:06:28.869201 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.869162    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120"} err="failed to get container status \"a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120\": rpc error: code = NotFound desc = could not find container \"a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120\": container with ID starting with a8ed2c10287b86c0cb22a03f5ad6448339fc68c9aebf4f7f15e242a570a24120 not found: ID does not exist"
Jul 13 19:06:28.869201 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.869180    2581 scope.go:117] "RemoveContainer" containerID="b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d"
Jul 13 19:06:28.869407 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:06:28.869389    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d\": container with ID starting with b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d not found: ID does not exist" containerID="b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d"
Jul 13 19:06:28.869446 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.869414    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d"} err="failed to get container status \"b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d\": rpc error: code = NotFound desc = could not find container \"b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d\": container with ID starting with b154e928e6bd0a06e7c27a21acbc748b6480cd1a6ae5f9621ff865639993119d not found: ID does not exist"
Jul 13 19:06:28.869446 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.869430    2581 scope.go:117] "RemoveContainer" containerID="9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd"
Jul 13 19:06:28.869638 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:06:28.869624    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd\": container with ID starting with 9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd not found: ID does not exist" containerID="9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd"
Jul 13 19:06:28.869677 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.869643    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd"} err="failed to get container status \"9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd\": rpc error: code = NotFound desc = could not find container \"9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd\": container with ID starting with 9d18056dc6056082c0330a0093d660dd914bc52e06cd6529fdd3530ca4af38bd not found: ID does not exist"
Jul 13 19:06:28.875182 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:28.875156    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw"]
Jul 13 19:06:29.604169 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:29.604134    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8461ba0-2951-4344-a855-6a37faceee62" path="/var/lib/kubelet/pods/b8461ba0-2951-4344-a855-6a37faceee62/volumes"
Jul 13 19:06:41.257210 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:41.257175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:41.718570 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:41.718532    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" podUID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerName="main" containerID="cri-o://20c76622ad13696fe30daf6d277a308c1568df21eda3c930ff493ddfca135ad9" gracePeriod=45
Jul 13 19:06:41.909783 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:41.909742    2581 generic.go:356] "Generic (PLEG): container finished" podID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerID="20c76622ad13696fe30daf6d277a308c1568df21eda3c930ff493ddfca135ad9" exitCode=0
Jul 13 19:06:41.909950 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:41.909820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" event={"ID":"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df","Type":"ContainerDied","Data":"20c76622ad13696fe30daf6d277a308c1568df21eda3c930ff493ddfca135ad9"}
Jul 13 19:06:41.983780 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:41.983755    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:06:42.031240 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kserve-provision-location\") pod \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") "
Jul 13 19:06:42.031410 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-model-cache\") pod \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") "
Jul 13 19:06:42.031410 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-dshm\") pod \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") "
Jul 13 19:06:42.031410 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tmp-dir\") pod \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") "
Jul 13 19:06:42.031410 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031335    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-885gc\" (UniqueName: \"kubernetes.io/projected/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kube-api-access-885gc\") pod \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") "
Jul 13 19:06:42.031410 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-home\") pod \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") "
Jul 13 19:06:42.031690 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tls-certs\") pod \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\" (UID: \"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df\") "
Jul 13 19:06:42.031690 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" (UID: "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.031690 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" (UID: "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.031690 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031676    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.031690 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.031692    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.032325 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.032272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-home" (OuterVolumeSpecName: "home") pod "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" (UID: "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.034023 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.033996    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" (UID: "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:06:42.034137 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.034101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-dshm" (OuterVolumeSpecName: "dshm") pod "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" (UID: "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.034339 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.034313    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kube-api-access-885gc" (OuterVolumeSpecName: "kube-api-access-885gc") pod "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" (UID: "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df"). InnerVolumeSpecName "kube-api-access-885gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:06:42.088213 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.088168    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" (UID: "ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.132303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.132268    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.132303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.132296    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.132303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.132306    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.132303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.132314    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.132571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.132324    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-885gc\" (UniqueName: \"kubernetes.io/projected/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df-kube-api-access-885gc\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.916509 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.916473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f" event={"ID":"ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df","Type":"ContainerDied","Data":"e6512eb4b71e848252425e47b2b622e336e39ea403fdf429bd1ea8a45fc2b569"}
Jul 13 19:06:42.916509 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.916514    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"
Jul 13 19:06:42.917013 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.916549    2581 scope.go:117] "RemoveContainer" containerID="20c76622ad13696fe30daf6d277a308c1568df21eda3c930ff493ddfca135ad9"
Jul 13 19:06:42.925495 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.925475    2581 scope.go:117] "RemoveContainer" containerID="c750b83aedaf25630411eff2a3b9c4315ba531743f1544ebe66b8c57a2baace6"
Jul 13 19:06:42.938258 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.938236    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"]
Jul 13 19:06:42.944959 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:42.944936    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f"]
Jul 13 19:06:43.603010 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:43.602968    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" path="/var/lib/kubelet/pods/ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df/volumes"
Jul 13 19:06:58.196442 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196411    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"]
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196686    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerName="storage-initializer"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196696    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerName="storage-initializer"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196712    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196718    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="storage-initializer"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196730    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="storage-initializer"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196737    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196741    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196752    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerName="main"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196757    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerName="main"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196763    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="tokenizer"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196768    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="tokenizer"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196840    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196847    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="tokenizer"
Jul 13 19:06:58.196902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.196853    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7b8723-9dc9-4f4f-aa9a-4e9f4098d2df" containerName="main"
Jul 13 19:06:58.199804 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.199768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.202935 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.202909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:06:58.202935 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.202909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:06:58.203149 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.202909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 13 19:06:58.203149 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.202914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:06:58.209468 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.209444    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"]
Jul 13 19:06:58.229271 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.229243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:58.260007 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.259976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.260161 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.260016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cddcd08-4b74-4456-9884-69c630b920f7-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.260161 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.260040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.260161 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.260106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.260161 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.260145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.260303 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.260166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nnk\" (UniqueName: \"kubernetes.io/projected/5cddcd08-4b74-4456-9884-69c630b920f7-kube-api-access-h9nnk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361268 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361268 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361507 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nnk\" (UniqueName: \"kubernetes.io/projected/5cddcd08-4b74-4456-9884-69c630b920f7-kube-api-access-h9nnk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361507 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361507 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cddcd08-4b74-4456-9884-69c630b920f7-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361507 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361760 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361905 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.361971 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.361925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.363778 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.363755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.364044 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.364026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cddcd08-4b74-4456-9884-69c630b920f7-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.369697 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.369666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nnk\" (UniqueName: \"kubernetes.io/projected/5cddcd08-4b74-4456-9884-69c630b920f7-kube-api-access-h9nnk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.469282 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.469249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:58.510060 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.510027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:06:58.649259 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.649235    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"]
Jul 13 19:06:58.651609 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:06:58.651583    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cddcd08_4b74_4456_9884_69c630b920f7.slice/crio-ee4b407bf8028133031b6881d14bb210cc2cf30b9dc7225cc1e357a31acbdf61 WatchSource:0}: Error finding container ee4b407bf8028133031b6881d14bb210cc2cf30b9dc7225cc1e357a31acbdf61: Status 404 returned error can't find the container with id ee4b407bf8028133031b6881d14bb210cc2cf30b9dc7225cc1e357a31acbdf61
Jul 13 19:06:58.998799 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.998762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x" event={"ID":"5cddcd08-4b74-4456-9884-69c630b920f7","Type":"ContainerStarted","Data":"d86676197d4f678003e8e657d58926aa6eb36d05203b9254b03673d0d0dab976"}
Jul 13 19:06:58.998981 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:58.998833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x" event={"ID":"5cddcd08-4b74-4456-9884-69c630b920f7","Type":"ContainerStarted","Data":"ee4b407bf8028133031b6881d14bb210cc2cf30b9dc7225cc1e357a31acbdf61"}
Jul 13 19:06:59.020665 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:59.020590    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x" podStartSLOduration=1.020572504 podStartE2EDuration="1.020572504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:06:59.019380942 +0000 UTC m=+1055.919606334" watchObservedRunningTime="2026-07-13 19:06:59.020572504 +0000 UTC m=+1055.920797896"
Jul 13 19:06:59.511047 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:06:59.511013    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:07:08.510420 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:08.510374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:07:08.522828 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:08.522781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:07:09.059067 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:09.059040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:07:24.426814 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:24.426699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:07:25.884931 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:25.884896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:07:25.988677 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:25.988648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:07:33.003913 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:33.003883    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"]
Jul 13 19:07:33.027146 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:33.027117    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:07:33.103126 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:33.103090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:07:48.024885 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.024846    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x" podUID="5cddcd08-4b74-4456-9884-69c630b920f7" containerName="main" containerID="cri-o://d86676197d4f678003e8e657d58926aa6eb36d05203b9254b03673d0d0dab976" gracePeriod=45
Jul 13 19:07:48.048122 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.048081    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x" podUID="5cddcd08-4b74-4456-9884-69c630b920f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 13 19:07:48.234581 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.234552    2581 generic.go:356] "Generic (PLEG): container finished" podID="5cddcd08-4b74-4456-9884-69c630b920f7" containerID="d86676197d4f678003e8e657d58926aa6eb36d05203b9254b03673d0d0dab976" exitCode=0
Jul 13 19:07:48.234746 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.234626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x" event={"ID":"5cddcd08-4b74-4456-9884-69c630b920f7","Type":"ContainerDied","Data":"d86676197d4f678003e8e657d58926aa6eb36d05203b9254b03673d0d0dab976"}
Jul 13 19:07:48.275273 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.275216    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:07:48.363575 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.363541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cddcd08-4b74-4456-9884-69c630b920f7-tls-certs\") pod \"5cddcd08-4b74-4456-9884-69c630b920f7\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") "
Jul 13 19:07:48.363774 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.363595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-tmp-dir\") pod \"5cddcd08-4b74-4456-9884-69c630b920f7\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") "
Jul 13 19:07:48.363774 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.363617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9nnk\" (UniqueName: \"kubernetes.io/projected/5cddcd08-4b74-4456-9884-69c630b920f7-kube-api-access-h9nnk\") pod \"5cddcd08-4b74-4456-9884-69c630b920f7\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") "
Jul 13 19:07:48.363774 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.363637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-model-cache\") pod \"5cddcd08-4b74-4456-9884-69c630b920f7\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") "
Jul 13 19:07:48.363774 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.363668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-home\") pod \"5cddcd08-4b74-4456-9884-69c630b920f7\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") "
Jul 13 19:07:48.363774 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.363700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-dshm\") pod \"5cddcd08-4b74-4456-9884-69c630b920f7\" (UID: \"5cddcd08-4b74-4456-9884-69c630b920f7\") "
Jul 13 19:07:48.364075 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.363957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cddcd08-4b74-4456-9884-69c630b920f7" (UID: "5cddcd08-4b74-4456-9884-69c630b920f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:07:48.364075 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.364015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cddcd08-4b74-4456-9884-69c630b920f7" (UID: "5cddcd08-4b74-4456-9884-69c630b920f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:07:48.364075 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.364031    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-home" (OuterVolumeSpecName: "home") pod "5cddcd08-4b74-4456-9884-69c630b920f7" (UID: "5cddcd08-4b74-4456-9884-69c630b920f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:07:48.365960 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.365932    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cddcd08-4b74-4456-9884-69c630b920f7-kube-api-access-h9nnk" (OuterVolumeSpecName: "kube-api-access-h9nnk") pod "5cddcd08-4b74-4456-9884-69c630b920f7" (UID: "5cddcd08-4b74-4456-9884-69c630b920f7"). InnerVolumeSpecName "kube-api-access-h9nnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:07:48.365960 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.365946    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cddcd08-4b74-4456-9884-69c630b920f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cddcd08-4b74-4456-9884-69c630b920f7" (UID: "5cddcd08-4b74-4456-9884-69c630b920f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:07:48.366073 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.365972    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-dshm" (OuterVolumeSpecName: "dshm") pod "5cddcd08-4b74-4456-9884-69c630b920f7" (UID: "5cddcd08-4b74-4456-9884-69c630b920f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:07:48.464576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.464533    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:07:48.464576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.464567    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9nnk\" (UniqueName: \"kubernetes.io/projected/5cddcd08-4b74-4456-9884-69c630b920f7-kube-api-access-h9nnk\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:07:48.464576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.464577    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:07:48.464576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.464586    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:07:48.464874 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.464594    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cddcd08-4b74-4456-9884-69c630b920f7-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:07:48.464874 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:48.464603    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cddcd08-4b74-4456-9884-69c630b920f7-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:07:49.242011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:49.241979    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"
Jul 13 19:07:49.242495 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:49.241978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x" event={"ID":"5cddcd08-4b74-4456-9884-69c630b920f7","Type":"ContainerDied","Data":"ee4b407bf8028133031b6881d14bb210cc2cf30b9dc7225cc1e357a31acbdf61"}
Jul 13 19:07:49.242495 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:49.242131    2581 scope.go:117] "RemoveContainer" containerID="d86676197d4f678003e8e657d58926aa6eb36d05203b9254b03673d0d0dab976"
Jul 13 19:07:49.263478 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:49.263452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"]
Jul 13 19:07:49.269905 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:49.269881    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x"]
Jul 13 19:07:49.603602 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:07:49.603520    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cddcd08-4b74-4456-9884-69c630b920f7" path="/var/lib/kubelet/pods/5cddcd08-4b74-4456-9884-69c630b920f7/volumes"
Jul 13 19:08:01.996414 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:01.996379    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"]
Jul 13 19:08:01.996836 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:01.996724    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cddcd08-4b74-4456-9884-69c630b920f7" containerName="main"
Jul 13 19:08:01.996836 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:01.996734    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cddcd08-4b74-4456-9884-69c630b920f7" containerName="main"
Jul 13 19:08:01.996836 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:01.996814    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cddcd08-4b74-4456-9884-69c630b920f7" containerName="main"
Jul 13 19:08:01.996836 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:01.996824    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8461ba0-2951-4344-a855-6a37faceee62" containerName="main"
Jul 13 19:08:02.002724 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.000541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.004036 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.003948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:08:02.004036 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.003995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:08:02.004036 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.004005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:08:02.004294 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.003995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 13 19:08:02.010219 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.010200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"]
Jul 13 19:08:02.073073 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.073027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.073073 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.073075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97g6t\" (UniqueName: \"kubernetes.io/projected/1a4ed088-928f-4850-a6e5-d3619f031e31-kube-api-access-97g6t\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.073358 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.073131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.073358 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.073150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4ed088-928f-4850-a6e5-d3619f031e31-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.073358 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.073194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.073358 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.073236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174102 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174102 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4ed088-928f-4850-a6e5-d3619f031e31-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174368 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174368 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174368 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174368 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97g6t\" (UniqueName: \"kubernetes.io/projected/1a4ed088-928f-4850-a6e5-d3619f031e31-kube-api-access-97g6t\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.174667 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.174592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.176560 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.176540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.176830 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.176812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4ed088-928f-4850-a6e5-d3619f031e31-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.183839 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.183813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97g6t\" (UniqueName: \"kubernetes.io/projected/1a4ed088-928f-4850-a6e5-d3619f031e31-kube-api-access-97g6t\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.313526 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.313441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:02.428435 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.428400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:02.459993 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.459967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"]
Jul 13 19:08:02.462306 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:08:02.462273    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4ed088_928f_4850_a6e5_d3619f031e31.slice/crio-f8ce353f00e12ab30f64bd803e0754ab94454f5cda9d841d4b12157d7297850d WatchSource:0}: Error finding container f8ce353f00e12ab30f64bd803e0754ab94454f5cda9d841d4b12157d7297850d: Status 404 returned error can't find the container with id f8ce353f00e12ab30f64bd803e0754ab94454f5cda9d841d4b12157d7297850d
Jul 13 19:08:02.529088 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:02.529062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:03.310735 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:03.310701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj" event={"ID":"1a4ed088-928f-4850-a6e5-d3619f031e31","Type":"ContainerStarted","Data":"024faa174c7dc7894726be0f2cab3ed8c529a4f5773e1ac5f791d7af495b6e31"}
Jul 13 19:08:03.311134 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:03.310761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj" event={"ID":"1a4ed088-928f-4850-a6e5-d3619f031e31","Type":"ContainerStarted","Data":"f8ce353f00e12ab30f64bd803e0754ab94454f5cda9d841d4b12157d7297850d"}
Jul 13 19:08:03.314103 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:03.314083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:03.349179 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:03.349132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj" podStartSLOduration=2.349117347 podStartE2EDuration="2.349117347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:08:03.348177935 +0000 UTC m=+1120.248403326" watchObservedRunningTime="2026-07-13 19:08:03.349117347 +0000 UTC m=+1120.249342740"
Jul 13 19:08:10.019502 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:10.019466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:12.314191 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:12.314153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:12.326757 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:12.326726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:12.367166 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:12.367129    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:08:32.371083 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:32.371048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:32.441558 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:32.441523    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:41.630324 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:41.630290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:41.913136 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:41.913063    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"]
Jul 13 19:08:41.917503 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:41.917485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:41.920307 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:41.920282    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-nkzdk\""
Jul 13 19:08:41.920307 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:41.920299    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 13 19:08:41.928938 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:41.928915    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"]
Jul 13 19:08:41.953843 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:41.953772    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:42.003382 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.003344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/52299f92-01cf-49be-8b75-4c0de732a5b7-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.003550 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.003396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhvdp\" (UniqueName: \"kubernetes.io/projected/52299f92-01cf-49be-8b75-4c0de732a5b7-kube-api-access-xhvdp\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.003550 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.003465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52299f92-01cf-49be-8b75-4c0de732a5b7-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.103873 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.103833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/52299f92-01cf-49be-8b75-4c0de732a5b7-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.104055 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.103897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhvdp\" (UniqueName: \"kubernetes.io/projected/52299f92-01cf-49be-8b75-4c0de732a5b7-kube-api-access-xhvdp\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.104055 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.103944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52299f92-01cf-49be-8b75-4c0de732a5b7-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.104257 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.104237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/52299f92-01cf-49be-8b75-4c0de732a5b7-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.106653 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.106628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52299f92-01cf-49be-8b75-4c0de732a5b7-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.113962 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.113937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhvdp\" (UniqueName: \"kubernetes.io/projected/52299f92-01cf-49be-8b75-4c0de732a5b7-kube-api-access-xhvdp\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.226385 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.226342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:42.380728 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.380670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"]
Jul 13 19:08:42.382941 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:08:42.382912    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52299f92_01cf_49be_8b75_4c0de732a5b7.slice/crio-e073486796321ee5a2d9a2ed203d559b61d015a0dd90e6485cb0c437ac43c678 WatchSource:0}: Error finding container e073486796321ee5a2d9a2ed203d559b61d015a0dd90e6485cb0c437ac43c678: Status 404 returned error can't find the container with id e073486796321ee5a2d9a2ed203d559b61d015a0dd90e6485cb0c437ac43c678
Jul 13 19:08:42.502071 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.501983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" event={"ID":"52299f92-01cf-49be-8b75-4c0de732a5b7","Type":"ContainerStarted","Data":"6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4"}
Jul 13 19:08:42.502071 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.502043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" event={"ID":"52299f92-01cf-49be-8b75-4c0de732a5b7","Type":"ContainerStarted","Data":"e073486796321ee5a2d9a2ed203d559b61d015a0dd90e6485cb0c437ac43c678"}
Jul 13 19:08:42.502245 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:42.502083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:08:47.380276 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:47.380193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:08:57.097738 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:08:57.097700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:03.442766 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:03.442725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:04.548176 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:04.548140    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:09.220457 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:09.220425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:13.508019 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:13.507984    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:09:13.527343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:13.527289    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" podStartSLOduration=32.527274869 podStartE2EDuration="32.527274869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:08:42.561722196 +0000 UTC m=+1159.461947588" watchObservedRunningTime="2026-07-13 19:09:13.527274869 +0000 UTC m=+1190.427500260"
Jul 13 19:09:15.317031 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.316996    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"]
Jul 13 19:09:15.323064 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.323028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:15.326377 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:09:15.326344    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:09:15.326377 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:09:15.326377 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:09:15.326377 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4" cmd=["/bin/sleep","15"]
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.326399    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" message="" err=<
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:09:15.326416    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" podUID="52299f92-01cf-49be-8b75-4c0de732a5b7" containerName="main" containerID="cri-o://6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4"
Jul 13 19:09:15.326614 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.326456    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" podUID="52299f92-01cf-49be-8b75-4c0de732a5b7" containerName="main" containerID="cri-o://6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4" gracePeriod=60
Jul 13 19:09:15.392572 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.392542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:15.576469 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.576408    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:09:15.661138 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.658538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52299f92-01cf-49be-8b75-4c0de732a5b7-tls-certs\") pod \"52299f92-01cf-49be-8b75-4c0de732a5b7\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") "
Jul 13 19:09:15.661138 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.658636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/52299f92-01cf-49be-8b75-4c0de732a5b7-tokenizer-uds\") pod \"52299f92-01cf-49be-8b75-4c0de732a5b7\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") "
Jul 13 19:09:15.661138 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.658676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhvdp\" (UniqueName: \"kubernetes.io/projected/52299f92-01cf-49be-8b75-4c0de732a5b7-kube-api-access-xhvdp\") pod \"52299f92-01cf-49be-8b75-4c0de732a5b7\" (UID: \"52299f92-01cf-49be-8b75-4c0de732a5b7\") "
Jul 13 19:09:15.661450 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.661187    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52299f92-01cf-49be-8b75-4c0de732a5b7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "52299f92-01cf-49be-8b75-4c0de732a5b7" (UID: "52299f92-01cf-49be-8b75-4c0de732a5b7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:15.662258 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.662226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52299f92-01cf-49be-8b75-4c0de732a5b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52299f92-01cf-49be-8b75-4c0de732a5b7" (UID: "52299f92-01cf-49be-8b75-4c0de732a5b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:09:15.663077 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.663048    2581 generic.go:356] "Generic (PLEG): container finished" podID="52299f92-01cf-49be-8b75-4c0de732a5b7" containerID="6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4" exitCode=0
Jul 13 19:09:15.663214 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.663112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"
Jul 13 19:09:15.663214 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.663133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" event={"ID":"52299f92-01cf-49be-8b75-4c0de732a5b7","Type":"ContainerDied","Data":"6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4"}
Jul 13 19:09:15.663326 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.663221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq" event={"ID":"52299f92-01cf-49be-8b75-4c0de732a5b7","Type":"ContainerDied","Data":"e073486796321ee5a2d9a2ed203d559b61d015a0dd90e6485cb0c437ac43c678"}
Jul 13 19:09:15.663326 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.663245    2581 scope.go:117] "RemoveContainer" containerID="6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4"
Jul 13 19:09:15.663438 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.663343    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52299f92-01cf-49be-8b75-4c0de732a5b7-kube-api-access-xhvdp" (OuterVolumeSpecName: "kube-api-access-xhvdp") pod "52299f92-01cf-49be-8b75-4c0de732a5b7" (UID: "52299f92-01cf-49be-8b75-4c0de732a5b7"). InnerVolumeSpecName "kube-api-access-xhvdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:09:15.677115 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.677094    2581 scope.go:117] "RemoveContainer" containerID="6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4"
Jul 13 19:09:15.677369 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:09:15.677350    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4\": container with ID starting with 6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4 not found: ID does not exist" containerID="6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4"
Jul 13 19:09:15.677434 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.677378    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4"} err="failed to get container status \"6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4\": rpc error: code = NotFound desc = could not find container \"6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4\": container with ID starting with 6ce9fd85ed967aeb25c2688ebc2cdca2e0f092e71ef50257277ec9469458c5b4 not found: ID does not exist"
Jul 13 19:09:15.760174 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.760140    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/52299f92-01cf-49be-8b75-4c0de732a5b7-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:15.760174 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.760165    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhvdp\" (UniqueName: \"kubernetes.io/projected/52299f92-01cf-49be-8b75-4c0de732a5b7-kube-api-access-xhvdp\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:15.760174 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.760176    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52299f92-01cf-49be-8b75-4c0de732a5b7-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:15.985941 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.985914    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"]
Jul 13 19:09:15.990119 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:15.990092    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq"]
Jul 13 19:09:17.602966 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:17.602937    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52299f92-01cf-49be-8b75-4c0de732a5b7" path="/var/lib/kubelet/pods/52299f92-01cf-49be-8b75-4c0de732a5b7/volumes"
Jul 13 19:09:23.603065 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:23.603037    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:09:23.603516 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:23.603123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:09:35.670555 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:35.670522    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"]
Jul 13 19:09:36.061210 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:36.061174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:36.162141 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:36.162107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:50.691077 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:50.691027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj" podUID="1a4ed088-928f-4850-a6e5-d3619f031e31" containerName="main" containerID="cri-o://024faa174c7dc7894726be0f2cab3ed8c529a4f5773e1ac5f791d7af495b6e31" gracePeriod=45
Jul 13 19:09:50.825632 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:50.825594    2581 generic.go:356] "Generic (PLEG): container finished" podID="1a4ed088-928f-4850-a6e5-d3619f031e31" containerID="024faa174c7dc7894726be0f2cab3ed8c529a4f5773e1ac5f791d7af495b6e31" exitCode=0
Jul 13 19:09:50.825868 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:50.825661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj" event={"ID":"1a4ed088-928f-4850-a6e5-d3619f031e31","Type":"ContainerDied","Data":"024faa174c7dc7894726be0f2cab3ed8c529a4f5773e1ac5f791d7af495b6e31"}
Jul 13 19:09:50.944820 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:50.944742    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:09:51.057771 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.057727    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-dshm\") pod \"1a4ed088-928f-4850-a6e5-d3619f031e31\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") "
Jul 13 19:09:51.057771 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.057767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-tmp-dir\") pod \"1a4ed088-928f-4850-a6e5-d3619f031e31\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") "
Jul 13 19:09:51.058043 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.057830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-home\") pod \"1a4ed088-928f-4850-a6e5-d3619f031e31\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") "
Jul 13 19:09:51.058043 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.057857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-model-cache\") pod \"1a4ed088-928f-4850-a6e5-d3619f031e31\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") "
Jul 13 19:09:51.058043 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.057898    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97g6t\" (UniqueName: \"kubernetes.io/projected/1a4ed088-928f-4850-a6e5-d3619f031e31-kube-api-access-97g6t\") pod \"1a4ed088-928f-4850-a6e5-d3619f031e31\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") "
Jul 13 19:09:51.058043 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.057926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4ed088-928f-4850-a6e5-d3619f031e31-tls-certs\") pod \"1a4ed088-928f-4850-a6e5-d3619f031e31\" (UID: \"1a4ed088-928f-4850-a6e5-d3619f031e31\") "
Jul 13 19:09:51.058259 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.058106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a4ed088-928f-4850-a6e5-d3619f031e31" (UID: "1a4ed088-928f-4850-a6e5-d3619f031e31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:51.058259 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.058158    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-home" (OuterVolumeSpecName: "home") pod "1a4ed088-928f-4850-a6e5-d3619f031e31" (UID: "1a4ed088-928f-4850-a6e5-d3619f031e31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:51.058259 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.058179    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a4ed088-928f-4850-a6e5-d3619f031e31" (UID: "1a4ed088-928f-4850-a6e5-d3619f031e31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:51.058400 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.058263    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:51.060074 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.060050    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-dshm" (OuterVolumeSpecName: "dshm") pod "1a4ed088-928f-4850-a6e5-d3619f031e31" (UID: "1a4ed088-928f-4850-a6e5-d3619f031e31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:51.060185 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.060110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4ed088-928f-4850-a6e5-d3619f031e31-kube-api-access-97g6t" (OuterVolumeSpecName: "kube-api-access-97g6t") pod "1a4ed088-928f-4850-a6e5-d3619f031e31" (UID: "1a4ed088-928f-4850-a6e5-d3619f031e31"). InnerVolumeSpecName "kube-api-access-97g6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:09:51.060224 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.060180    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4ed088-928f-4850-a6e5-d3619f031e31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a4ed088-928f-4850-a6e5-d3619f031e31" (UID: "1a4ed088-928f-4850-a6e5-d3619f031e31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:09:51.159199 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.159163    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97g6t\" (UniqueName: \"kubernetes.io/projected/1a4ed088-928f-4850-a6e5-d3619f031e31-kube-api-access-97g6t\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:51.159199 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.159191    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a4ed088-928f-4850-a6e5-d3619f031e31-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:51.159199 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.159202    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:51.159199 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.159210    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:51.159472 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.159217    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a4ed088-928f-4850-a6e5-d3619f031e31-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:09:51.831602 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.831568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj" event={"ID":"1a4ed088-928f-4850-a6e5-d3619f031e31","Type":"ContainerDied","Data":"f8ce353f00e12ab30f64bd803e0754ab94454f5cda9d841d4b12157d7297850d"}
Jul 13 19:09:51.832047 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.831645    2581 scope.go:117] "RemoveContainer" containerID="024faa174c7dc7894726be0f2cab3ed8c529a4f5773e1ac5f791d7af495b6e31"
Jul 13 19:09:51.832047 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.831656    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"
Jul 13 19:09:51.851579 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.851546    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"]
Jul 13 19:09:51.856228 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:51.856203    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj"]
Jul 13 19:09:53.608806 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:53.608744    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4ed088-928f-4850-a6e5-d3619f031e31" path="/var/lib/kubelet/pods/1a4ed088-928f-4850-a6e5-d3619f031e31/volumes"
Jul 13 19:09:53.642310 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:09:53.642280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:10:07.408081 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.408044    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"]
Jul 13 19:10:07.408467 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.408334    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52299f92-01cf-49be-8b75-4c0de732a5b7" containerName="main"
Jul 13 19:10:07.408467 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.408344    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52299f92-01cf-49be-8b75-4c0de732a5b7" containerName="main"
Jul 13 19:10:07.408467 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.408364    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4ed088-928f-4850-a6e5-d3619f031e31" containerName="main"
Jul 13 19:10:07.408467 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.408370    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4ed088-928f-4850-a6e5-d3619f031e31" containerName="main"
Jul 13 19:10:07.408467 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.408426    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4ed088-928f-4850-a6e5-d3619f031e31" containerName="main"
Jul 13 19:10:07.408467 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.408432    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="52299f92-01cf-49be-8b75-4c0de732a5b7" containerName="main"
Jul 13 19:10:07.412494 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.412475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.416015 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.415993    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:10:07.416150 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.416015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 13 19:10:07.416150 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.416015    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:10:07.416150 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.416065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:10:07.423242 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.423219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"]
Jul 13 19:10:07.469073 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.469029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.469073 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.469079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.469284 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.469123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwjmk\" (UniqueName: \"kubernetes.io/projected/c63e7ead-010b-462b-9110-daa88af18ff4-kube-api-access-gwjmk\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.469284 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.469147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63e7ead-010b-462b-9110-daa88af18ff4-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.469284 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.469203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.469284 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.469238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.482211 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.482180    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:10:07.570308 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570308 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570519 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjmk\" (UniqueName: \"kubernetes.io/projected/c63e7ead-010b-462b-9110-daa88af18ff4-kube-api-access-gwjmk\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570519 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63e7ead-010b-462b-9110-daa88af18ff4-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570519 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570697 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570777 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570882 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.570935 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.570905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.572672 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.572652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.573065 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.573045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63e7ead-010b-462b-9110-daa88af18ff4-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.578844 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.578823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwjmk\" (UniqueName: \"kubernetes.io/projected/c63e7ead-010b-462b-9110-daa88af18ff4-kube-api-access-gwjmk\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.723168 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.723125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:07.742227 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.742186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:10:07.867165 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.867138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"]
Jul 13 19:10:07.869167 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:10:07.869130    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63e7ead_010b_462b_9110_daa88af18ff4.slice/crio-8f23e7ceb7ba0a381724e315d85fab63a37ec407cbc75bf73360890ab3543304 WatchSource:0}: Error finding container 8f23e7ceb7ba0a381724e315d85fab63a37ec407cbc75bf73360890ab3543304: Status 404 returned error can't find the container with id 8f23e7ceb7ba0a381724e315d85fab63a37ec407cbc75bf73360890ab3543304
Jul 13 19:10:07.908271 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:07.908230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx" event={"ID":"c63e7ead-010b-462b-9110-daa88af18ff4","Type":"ContainerStarted","Data":"8f23e7ceb7ba0a381724e315d85fab63a37ec407cbc75bf73360890ab3543304"}
Jul 13 19:10:08.914777 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:08.914743    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx" event={"ID":"c63e7ead-010b-462b-9110-daa88af18ff4","Type":"ContainerStarted","Data":"e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42"}
Jul 13 19:10:08.933345 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:08.933298    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx" podStartSLOduration=1.933283391 podStartE2EDuration="1.933283391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:10:08.931128522 +0000 UTC m=+1245.831353925" watchObservedRunningTime="2026-07-13 19:10:08.933283391 +0000 UTC m=+1245.833508782"
Jul 13 19:10:09.723355 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:09.723325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:17.724060 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:17.723964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:17.736413 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:17.736392    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:17.968827 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:17.968779    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:10:19.397631 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:10:19.397600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:05.552169 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:05.552128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:05.762007 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:05.761910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:41.298293 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.298259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:41.652161 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.652067    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"]
Jul 13 19:11:41.655563 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.655543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.659663 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.659641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 13 19:11:41.659827 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.659698    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-s2gqm\""
Jul 13 19:11:41.660005 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.659977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:11:41.690564 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.690530    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"]
Jul 13 19:11:41.744968 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.744936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.745128 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.744984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvjp\" (UniqueName: \"kubernetes.io/projected/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kube-api-access-gvvjp\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.745128 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.745052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.745128 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.745079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.845737 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.845692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvjp\" (UniqueName: \"kubernetes.io/projected/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kube-api-access-gvvjp\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.845957 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.845757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.845957 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.845808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.845957 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.845880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.846226 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.846200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.846375 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.846243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.848861 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.848837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.856999 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.856970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvjp\" (UniqueName: \"kubernetes.io/projected/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kube-api-access-gvvjp\") pod \"auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:41.965596 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:41.965561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:42.132495 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:42.132457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"]
Jul 13 19:11:42.138878 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:11:42.138847    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fb725a_bc81_4c1b_a0e9_ebe061a8b62e.slice/crio-4090261f9801a1b2e14d7bf8614574275fe41d6468394e3a2fe6aac70c18961a WatchSource:0}: Error finding container 4090261f9801a1b2e14d7bf8614574275fe41d6468394e3a2fe6aac70c18961a: Status 404 returned error can't find the container with id 4090261f9801a1b2e14d7bf8614574275fe41d6468394e3a2fe6aac70c18961a
Jul 13 19:11:42.360605 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:42.360510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" event={"ID":"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e","Type":"ContainerStarted","Data":"d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce"}
Jul 13 19:11:42.360605 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:42.360558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" event={"ID":"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e","Type":"ContainerStarted","Data":"4090261f9801a1b2e14d7bf8614574275fe41d6468394e3a2fe6aac70c18961a"}
Jul 13 19:11:43.178440 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:43.178401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:43.366925 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:43.366892    2581 generic.go:356] "Generic (PLEG): container finished" podID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerID="d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce" exitCode=0
Jul 13 19:11:43.367296 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:43.366935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" event={"ID":"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e","Type":"ContainerDied","Data":"d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce"}
Jul 13 19:11:44.374777 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:44.374722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" event={"ID":"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e","Type":"ContainerStarted","Data":"db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461"}
Jul 13 19:11:44.375249 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:44.374844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:11:44.403585 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:44.403515    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" podStartSLOduration=2.671381747 podStartE2EDuration="3.403475804s" totalImagesPullingTime="0s" totalInitContainerRuntime="732.094057ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:11:44.403323537 +0000 UTC m=+1341.303548932" watchObservedRunningTime="2026-07-13 19:11:44.403475804 +0000 UTC m=+1341.303701193"
Jul 13 19:11:56.842090 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:56.842050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:56.956686 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:11:56.956652    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:15.380953 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:15.380924    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:12:18.030239 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:18.030207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:29.922197 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:29.922154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:30.374609 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:30.374579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:32.994933 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:32.994897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:33.102499 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:33.102468    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:49.371057 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:49.371021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:49.439551 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:49.439513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:52.776015 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:52.775977    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:52.783475 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:52.783448    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"]
Jul 13 19:12:52.848256 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:12:52.848213    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:07.804815 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:07.804737    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx" podUID="c63e7ead-010b-462b-9110-daa88af18ff4" containerName="main" containerID="cri-o://e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42" gracePeriod=45
Jul 13 19:13:07.958590 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:07.958540    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx" podUID="c63e7ead-010b-462b-9110-daa88af18ff4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 13 19:13:08.051838 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.051809    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:13:08.164413 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-model-cache\") pod \"c63e7ead-010b-462b-9110-daa88af18ff4\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") "
Jul 13 19:13:08.164413 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-home\") pod \"c63e7ead-010b-462b-9110-daa88af18ff4\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") "
Jul 13 19:13:08.164413 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-dshm\") pod \"c63e7ead-010b-462b-9110-daa88af18ff4\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") "
Jul 13 19:13:08.164710 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwjmk\" (UniqueName: \"kubernetes.io/projected/c63e7ead-010b-462b-9110-daa88af18ff4-kube-api-access-gwjmk\") pod \"c63e7ead-010b-462b-9110-daa88af18ff4\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") "
Jul 13 19:13:08.164710 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164480    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63e7ead-010b-462b-9110-daa88af18ff4-tls-certs\") pod \"c63e7ead-010b-462b-9110-daa88af18ff4\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") "
Jul 13 19:13:08.164710 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164614    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-home" (OuterVolumeSpecName: "home") pod "c63e7ead-010b-462b-9110-daa88af18ff4" (UID: "c63e7ead-010b-462b-9110-daa88af18ff4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:13:08.164710 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-tmp-dir\") pod \"c63e7ead-010b-462b-9110-daa88af18ff4\" (UID: \"c63e7ead-010b-462b-9110-daa88af18ff4\") "
Jul 13 19:13:08.164710 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164645    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c63e7ead-010b-462b-9110-daa88af18ff4" (UID: "c63e7ead-010b-462b-9110-daa88af18ff4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:13:08.164956 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c63e7ead-010b-462b-9110-daa88af18ff4" (UID: "c63e7ead-010b-462b-9110-daa88af18ff4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:13:08.164956 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164903    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:08.164956 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164921    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:08.164956 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.164932    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:08.166725 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.166703    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-dshm" (OuterVolumeSpecName: "dshm") pod "c63e7ead-010b-462b-9110-daa88af18ff4" (UID: "c63e7ead-010b-462b-9110-daa88af18ff4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:13:08.166838 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.166817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63e7ead-010b-462b-9110-daa88af18ff4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c63e7ead-010b-462b-9110-daa88af18ff4" (UID: "c63e7ead-010b-462b-9110-daa88af18ff4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:13:08.166838 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.166825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63e7ead-010b-462b-9110-daa88af18ff4-kube-api-access-gwjmk" (OuterVolumeSpecName: "kube-api-access-gwjmk") pod "c63e7ead-010b-462b-9110-daa88af18ff4" (UID: "c63e7ead-010b-462b-9110-daa88af18ff4"). InnerVolumeSpecName "kube-api-access-gwjmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:13:08.265860 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.265828    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63e7ead-010b-462b-9110-daa88af18ff4-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:08.265860 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.265856    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwjmk\" (UniqueName: \"kubernetes.io/projected/c63e7ead-010b-462b-9110-daa88af18ff4-kube-api-access-gwjmk\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:08.265860 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.265867    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63e7ead-010b-462b-9110-daa88af18ff4-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:08.768455 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.768408    2581 generic.go:356] "Generic (PLEG): container finished" podID="c63e7ead-010b-462b-9110-daa88af18ff4" containerID="e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42" exitCode=0
Jul 13 19:13:08.768455 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.768453    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"
Jul 13 19:13:08.768670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.768491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx" event={"ID":"c63e7ead-010b-462b-9110-daa88af18ff4","Type":"ContainerDied","Data":"e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42"}
Jul 13 19:13:08.768670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.768547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx" event={"ID":"c63e7ead-010b-462b-9110-daa88af18ff4","Type":"ContainerDied","Data":"8f23e7ceb7ba0a381724e315d85fab63a37ec407cbc75bf73360890ab3543304"}
Jul 13 19:13:08.768670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.768569    2581 scope.go:117] "RemoveContainer" containerID="e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42"
Jul 13 19:13:08.780243 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.780223    2581 scope.go:117] "RemoveContainer" containerID="e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42"
Jul 13 19:13:08.780521 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:13:08.780503    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42\": container with ID starting with e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42 not found: ID does not exist" containerID="e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42"
Jul 13 19:13:08.780572 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.780531    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42"} err="failed to get container status \"e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42\": rpc error: code = NotFound desc = could not find container \"e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42\": container with ID starting with e605cfe8dc56434b99f800cfc99dd37b9145cfb7a1419bb3ffc1ada181a37a42 not found: ID does not exist"
Jul 13 19:13:08.792316 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.792289    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"]
Jul 13 19:13:08.795919 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:08.795897    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx"]
Jul 13 19:13:09.604000 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:09.603967    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c63e7ead-010b-462b-9110-daa88af18ff4" path="/var/lib/kubelet/pods/c63e7ead-010b-462b-9110-daa88af18ff4/volumes"
Jul 13 19:13:19.038824 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.038707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:19.517382 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.517341    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"]
Jul 13 19:13:19.517695 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.517680    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63e7ead-010b-462b-9110-daa88af18ff4" containerName="main"
Jul 13 19:13:19.517746 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.517697    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63e7ead-010b-462b-9110-daa88af18ff4" containerName="main"
Jul 13 19:13:19.517848 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.517836    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c63e7ead-010b-462b-9110-daa88af18ff4" containerName="main"
Jul 13 19:13:19.522240 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.522223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.524418 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.524400    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-9f88j\""
Jul 13 19:13:19.524503 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.524412    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 13 19:13:19.531613 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.531583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"]
Jul 13 19:13:19.542131 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.542100    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:19.549951 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.549928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.550063 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.549962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.550063 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.549994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2jj\" (UniqueName: \"kubernetes.io/projected/b61db38b-9919-416b-a84c-06d301654c58-kube-api-access-xr2jj\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.550063 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.550012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61db38b-9919-416b-a84c-06d301654c58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.650818 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.650749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.650818 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.650819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.651034 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.650855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2jj\" (UniqueName: \"kubernetes.io/projected/b61db38b-9919-416b-a84c-06d301654c58-kube-api-access-xr2jj\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.651034 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.650873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61db38b-9919-416b-a84c-06d301654c58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.658061 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.651494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.658061 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.651312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.658061 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.654411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61db38b-9919-416b-a84c-06d301654c58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.667492 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.667467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2jj\" (UniqueName: \"kubernetes.io/projected/b61db38b-9919-416b-a84c-06d301654c58-kube-api-access-xr2jj\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.832499 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.832410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:19.975560 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:19.975530    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"]
Jul 13 19:13:19.977801 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:13:19.977751    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61db38b_9919_416b_a84c_06d301654c58.slice/crio-15ce0a7623c10b1e6d07677d2411ccd2c251a609af1146915760f52f402b125f WatchSource:0}: Error finding container 15ce0a7623c10b1e6d07677d2411ccd2c251a609af1146915760f52f402b125f: Status 404 returned error can't find the container with id 15ce0a7623c10b1e6d07677d2411ccd2c251a609af1146915760f52f402b125f
Jul 13 19:13:20.826526 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:20.826492    2581 generic.go:356] "Generic (PLEG): container finished" podID="b61db38b-9919-416b-a84c-06d301654c58" containerID="77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd" exitCode=0
Jul 13 19:13:20.826994 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:20.826544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" event={"ID":"b61db38b-9919-416b-a84c-06d301654c58","Type":"ContainerDied","Data":"77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd"}
Jul 13 19:13:20.826994 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:20.826578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" event={"ID":"b61db38b-9919-416b-a84c-06d301654c58","Type":"ContainerStarted","Data":"15ce0a7623c10b1e6d07677d2411ccd2c251a609af1146915760f52f402b125f"}
Jul 13 19:13:21.832690 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:21.832656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" event={"ID":"b61db38b-9919-416b-a84c-06d301654c58","Type":"ContainerStarted","Data":"e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305"}
Jul 13 19:13:21.833157 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:21.832826    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:13:21.857130 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:21.857079    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" podStartSLOduration=2.115520258 podStartE2EDuration="2.857065149s" totalImagesPullingTime="0s" totalInitContainerRuntime="741.544891ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:13:21.855294889 +0000 UTC m=+1438.755520295" watchObservedRunningTime="2026-07-13 19:13:21.857065149 +0000 UTC m=+1438.757290541"
Jul 13 19:13:33.216535 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:33.216507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:47.397948 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.397648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:47.401021 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.400991    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"]
Jul 13 19:13:47.410177 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:13:47.410139    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:13:47.410177 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:13:47.410177 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:13:47.410177 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461" cmd=["/bin/sleep","15"]
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.410193    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" message="" err=<
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:13:47.410207    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerName="main" containerID="cri-o://db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461"
Jul 13 19:13:47.410405 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.410253    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerName="main" containerID="cri-o://db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461" gracePeriod=60
Jul 13 19:13:47.474152 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.474120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:47.661779 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.661713    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:13:47.782738 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.782693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvvjp\" (UniqueName: \"kubernetes.io/projected/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kube-api-access-gvvjp\") pod \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") "
Jul 13 19:13:47.782738 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.782752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tls-certs\") pod \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") "
Jul 13 19:13:47.783044 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.782774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kserve-provision-location\") pod \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") "
Jul 13 19:13:47.783044 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.782917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tokenizer-uds\") pod \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\" (UID: \"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e\") "
Jul 13 19:13:47.783154 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.783135    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" (UID: "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:13:47.783500 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.783397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" (UID: "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:13:47.785061 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.785037    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" (UID: "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:13:47.785163 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.785113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kube-api-access-gvvjp" (OuterVolumeSpecName: "kube-api-access-gvvjp") pod "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" (UID: "62fb725a-bc81-4c1b-a0e9-ebe061a8b62e"). InnerVolumeSpecName "kube-api-access-gvvjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:13:47.884193 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.884156    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:47.884193 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.884187    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvvjp\" (UniqueName: \"kubernetes.io/projected/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kube-api-access-gvvjp\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:47.884193 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.884200    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:47.884430 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.884213    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:13:47.962309 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.962274    2581 generic.go:356] "Generic (PLEG): container finished" podID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerID="db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461" exitCode=0
Jul 13 19:13:47.962481 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.962328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" event={"ID":"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e","Type":"ContainerDied","Data":"db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461"}
Jul 13 19:13:47.962481 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.962341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"
Jul 13 19:13:47.962481 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.962380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g" event={"ID":"62fb725a-bc81-4c1b-a0e9-ebe061a8b62e","Type":"ContainerDied","Data":"4090261f9801a1b2e14d7bf8614574275fe41d6468394e3a2fe6aac70c18961a"}
Jul 13 19:13:47.962481 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.962407    2581 scope.go:117] "RemoveContainer" containerID="db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461"
Jul 13 19:13:47.972099 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.972075    2581 scope.go:117] "RemoveContainer" containerID="d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce"
Jul 13 19:13:47.980622 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.980601    2581 scope.go:117] "RemoveContainer" containerID="db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461"
Jul 13 19:13:47.980942 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:13:47.980921    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461\": container with ID starting with db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461 not found: ID does not exist" containerID="db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461"
Jul 13 19:13:47.981011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.980950    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461"} err="failed to get container status \"db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461\": rpc error: code = NotFound desc = could not find container \"db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461\": container with ID starting with db760db07246130470aeace59bae35f9c37a8a9d08d3f13a73a552725a7f7461 not found: ID does not exist"
Jul 13 19:13:47.981011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.980967    2581 scope.go:117] "RemoveContainer" containerID="d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce"
Jul 13 19:13:47.981205 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:13:47.981188    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce\": container with ID starting with d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce not found: ID does not exist" containerID="d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce"
Jul 13 19:13:47.981246 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.981212    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce"} err="failed to get container status \"d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce\": rpc error: code = NotFound desc = could not find container \"d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce\": container with ID starting with d6d6667c3c673be1754e70aab707e2a31b26040300f7051f610534784ca227ce not found: ID does not exist"
Jul 13 19:13:47.986652 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.986630    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"]
Jul 13 19:13:47.989054 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:47.989033    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g"]
Jul 13 19:13:49.609572 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:49.609541    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" path="/var/lib/kubelet/pods/62fb725a-bc81-4c1b-a0e9-ebe061a8b62e/volumes"
Jul 13 19:13:52.839350 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:13:52.839320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:14:06.225527 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:14:06.225485    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:15.709252 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:14:15.709216    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:23.633842 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:14:23.633815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:14:23.635520 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:14:23.635499    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:15:17.415802 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.415750    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"]
Jul 13 19:15:17.424482 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:15:17.424446    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:15:17.424482 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:15:17.424482 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:15:17.424482 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305" cmd=["/bin/sleep","15"]
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.424496    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" message="" err=<
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:15:17.424507    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" podUID="b61db38b-9919-416b-a84c-06d301654c58" containerName="main" containerID="cri-o://e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305"
Jul 13 19:15:17.424709 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.424575    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" podUID="b61db38b-9919-416b-a84c-06d301654c58" containerName="main" containerID="cri-o://e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305" gracePeriod=60
Jul 13 19:15:17.674306 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.674249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:15:17.747769 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.747661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-kserve-provision-location\") pod \"b61db38b-9919-416b-a84c-06d301654c58\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") "
Jul 13 19:15:17.747769 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.747736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-tokenizer-uds\") pod \"b61db38b-9919-416b-a84c-06d301654c58\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") "
Jul 13 19:15:17.747769 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.747756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61db38b-9919-416b-a84c-06d301654c58-tls-certs\") pod \"b61db38b-9919-416b-a84c-06d301654c58\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") "
Jul 13 19:15:17.755971 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.747775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr2jj\" (UniqueName: \"kubernetes.io/projected/b61db38b-9919-416b-a84c-06d301654c58-kube-api-access-xr2jj\") pod \"b61db38b-9919-416b-a84c-06d301654c58\" (UID: \"b61db38b-9919-416b-a84c-06d301654c58\") "
Jul 13 19:15:17.755971 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.748030    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b61db38b-9919-416b-a84c-06d301654c58" (UID: "b61db38b-9919-416b-a84c-06d301654c58"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:17.755971 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.748320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b61db38b-9919-416b-a84c-06d301654c58" (UID: "b61db38b-9919-416b-a84c-06d301654c58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:17.755971 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.749910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61db38b-9919-416b-a84c-06d301654c58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b61db38b-9919-416b-a84c-06d301654c58" (UID: "b61db38b-9919-416b-a84c-06d301654c58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:15:17.755971 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.749992    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61db38b-9919-416b-a84c-06d301654c58-kube-api-access-xr2jj" (OuterVolumeSpecName: "kube-api-access-xr2jj") pod "b61db38b-9919-416b-a84c-06d301654c58" (UID: "b61db38b-9919-416b-a84c-06d301654c58"). InnerVolumeSpecName "kube-api-access-xr2jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:15:17.848441 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.848412    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:15:17.848441 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.848437    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61db38b-9919-416b-a84c-06d301654c58-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:15:17.848621 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.848452    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2jj\" (UniqueName: \"kubernetes.io/projected/b61db38b-9919-416b-a84c-06d301654c58-kube-api-access-xr2jj\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:15:17.848621 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:17.848465    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61db38b-9919-416b-a84c-06d301654c58-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:15:18.400402 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.400364    2581 generic.go:356] "Generic (PLEG): container finished" podID="b61db38b-9919-416b-a84c-06d301654c58" containerID="e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305" exitCode=0
Jul 13 19:15:18.400571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.400418    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"
Jul 13 19:15:18.400571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.400428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" event={"ID":"b61db38b-9919-416b-a84c-06d301654c58","Type":"ContainerDied","Data":"e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305"}
Jul 13 19:15:18.400571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.400478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97" event={"ID":"b61db38b-9919-416b-a84c-06d301654c58","Type":"ContainerDied","Data":"15ce0a7623c10b1e6d07677d2411ccd2c251a609af1146915760f52f402b125f"}
Jul 13 19:15:18.400571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.400508    2581 scope.go:117] "RemoveContainer" containerID="e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305"
Jul 13 19:15:18.409885 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.409869    2581 scope.go:117] "RemoveContainer" containerID="77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd"
Jul 13 19:15:18.417527 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.417507    2581 scope.go:117] "RemoveContainer" containerID="e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305"
Jul 13 19:15:18.417856 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:15:18.417806    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305\": container with ID starting with e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305 not found: ID does not exist" containerID="e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305"
Jul 13 19:15:18.417900 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.417846    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305"} err="failed to get container status \"e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305\": rpc error: code = NotFound desc = could not find container \"e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305\": container with ID starting with e05931de905f44a191527c9520a7d3cc323c5dd70f767ded91df264ffbc54305 not found: ID does not exist"
Jul 13 19:15:18.417900 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.417871    2581 scope.go:117] "RemoveContainer" containerID="77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd"
Jul 13 19:15:18.418134 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:15:18.418117    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd\": container with ID starting with 77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd not found: ID does not exist" containerID="77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd"
Jul 13 19:15:18.418173 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.418139    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd"} err="failed to get container status \"77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd\": rpc error: code = NotFound desc = could not find container \"77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd\": container with ID starting with 77078ebb8df7ff7fcedc5f56e5e1873654c08b5617a22c1fccb25d901fe4decd not found: ID does not exist"
Jul 13 19:15:18.432241 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.432220    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"]
Jul 13 19:15:18.439857 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:18.439836    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97"]
Jul 13 19:15:19.287090 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:19.287061    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:19.369545 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:19.369516    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:19.603459 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:19.603371    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61db38b-9919-416b-a84c-06d301654c58" path="/var/lib/kubelet/pods/b61db38b-9919-416b-a84c-06d301654c58/volumes"
Jul 13 19:15:48.115817 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.115756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:48.680608 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680571    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"]
Jul 13 19:15:48.680896 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680882    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61db38b-9919-416b-a84c-06d301654c58" containerName="main"
Jul 13 19:15:48.680896 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680897    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61db38b-9919-416b-a84c-06d301654c58" containerName="main"
Jul 13 19:15:48.680991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680917    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerName="storage-initializer"
Jul 13 19:15:48.680991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680923    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerName="storage-initializer"
Jul 13 19:15:48.680991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680931    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerName="main"
Jul 13 19:15:48.680991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680936    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerName="main"
Jul 13 19:15:48.680991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680941    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61db38b-9919-416b-a84c-06d301654c58" containerName="storage-initializer"
Jul 13 19:15:48.680991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680946    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61db38b-9919-416b-a84c-06d301654c58" containerName="storage-initializer"
Jul 13 19:15:48.680991 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680991    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fb725a-bc81-4c1b-a0e9-ebe061a8b62e" containerName="main"
Jul 13 19:15:48.681229 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.680998    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61db38b-9919-416b-a84c-06d301654c58" containerName="main"
Jul 13 19:15:48.685002 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.684979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.687421 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.687386    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-xzf8g\""
Jul 13 19:15:48.687421 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.687401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:15:48.688064 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.688037    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:15:48.688182 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.688117    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:15:48.688182 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.688122    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:15:48.688308 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.688216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 13 19:15:48.695570 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.695550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"]
Jul 13 19:15:48.721164 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.721128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:48.786669 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.786635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11175152-4faf-48f5-a990-ca1782914956-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.786819 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.786674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddmv\" (UniqueName: \"kubernetes.io/projected/11175152-4faf-48f5-a990-ca1782914956-kube-api-access-jddmv\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.786819 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.786770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.786906 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.786834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.887507 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.887476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.887700 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.887521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11175152-4faf-48f5-a990-ca1782914956-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.887700 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.887538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddmv\" (UniqueName: \"kubernetes.io/projected/11175152-4faf-48f5-a990-ca1782914956-kube-api-access-jddmv\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.887700 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.887590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.887980 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.887952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.887980 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.887965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.890236 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.890215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11175152-4faf-48f5-a990-ca1782914956-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.895665 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.895639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddmv\" (UniqueName: \"kubernetes.io/projected/11175152-4faf-48f5-a990-ca1782914956-kube-api-access-jddmv\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:48.995430 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:48.995393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:49.129721 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:49.129696    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"]
Jul 13 19:15:49.132411 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:15:49.132374    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11175152_4faf_48f5_a990_ca1782914956.slice/crio-b1b980410ae87a07a90fb339b05b3facfef7f6bd156ba54c7182c218bb2545a8 WatchSource:0}: Error finding container b1b980410ae87a07a90fb339b05b3facfef7f6bd156ba54c7182c218bb2545a8: Status 404 returned error can't find the container with id b1b980410ae87a07a90fb339b05b3facfef7f6bd156ba54c7182c218bb2545a8
Jul 13 19:15:49.545764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:49.545722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" event={"ID":"11175152-4faf-48f5-a990-ca1782914956","Type":"ContainerStarted","Data":"eab17495e6690e613e3df1e4adeaf024931a099c6f7a1be314fc06dccfc5b8e1"}
Jul 13 19:15:49.545972 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:49.545781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" event={"ID":"11175152-4faf-48f5-a990-ca1782914956","Type":"ContainerStarted","Data":"b1b980410ae87a07a90fb339b05b3facfef7f6bd156ba54c7182c218bb2545a8"}
Jul 13 19:15:50.554040 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:50.552603    2581 generic.go:356] "Generic (PLEG): container finished" podID="11175152-4faf-48f5-a990-ca1782914956" containerID="eab17495e6690e613e3df1e4adeaf024931a099c6f7a1be314fc06dccfc5b8e1" exitCode=0
Jul 13 19:15:50.554040 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:50.552673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" event={"ID":"11175152-4faf-48f5-a990-ca1782914956","Type":"ContainerDied","Data":"eab17495e6690e613e3df1e4adeaf024931a099c6f7a1be314fc06dccfc5b8e1"}
Jul 13 19:15:51.561517 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:51.561476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" event={"ID":"11175152-4faf-48f5-a990-ca1782914956","Type":"ContainerStarted","Data":"a1c65586d1f844ffeaf49e23305009389be2abc1b0878d71d7e22643b4e44f08"}
Jul 13 19:15:51.562002 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:51.561601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:15:51.581358 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:15:51.581298    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" podStartSLOduration=2.83036709 podStartE2EDuration="3.581282967s" totalImagesPullingTime="0s" totalInitContainerRuntime="750.915877ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:15:51.578925485 +0000 UTC m=+1588.479150871" watchObservedRunningTime="2026-07-13 19:15:51.581282967 +0000 UTC m=+1588.481508406"
Jul 13 19:16:02.794047 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:02.794015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:16:22.568197 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:22.568112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:16:25.460513 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:25.460463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:16:25.520585 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:25.520553    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:16:31.893173 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:31.893132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:16:56.323908 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.323864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:16:56.609895 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.609809    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"]
Jul 13 19:16:56.612817 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.612773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.615094 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.615074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-92gnr\""
Jul 13 19:16:56.615621 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.615602    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 13 19:16:56.628746 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.628722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"]
Jul 13 19:16:56.751268 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.751233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796644df-e26a-4421-bbca-f793b5cad3fc-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.751535 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.751509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.751660 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.751638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5fwg\" (UniqueName: \"kubernetes.io/projected/796644df-e26a-4421-bbca-f793b5cad3fc-kube-api-access-p5fwg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.751706 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.751674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.852452 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.852413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.852647 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.852464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5fwg\" (UniqueName: \"kubernetes.io/projected/796644df-e26a-4421-bbca-f793b5cad3fc-kube-api-access-p5fwg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.852647 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.852598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.852773 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.852663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796644df-e26a-4421-bbca-f793b5cad3fc-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.852856 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.852827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.852933 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.852914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.855208 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.855186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796644df-e26a-4421-bbca-f793b5cad3fc-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.860302 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.860237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5fwg\" (UniqueName: \"kubernetes.io/projected/796644df-e26a-4421-bbca-f793b5cad3fc-kube-api-access-p5fwg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:56.922675 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:56.922650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:57.059805 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:57.059756    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"]
Jul 13 19:16:57.062993 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:16:57.062969    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796644df_e26a_4421_bbca_f793b5cad3fc.slice/crio-818d760aeb79586c784eb67145d1ba318c410cbd72085e65aecbd9e11f998d9e WatchSource:0}: Error finding container 818d760aeb79586c784eb67145d1ba318c410cbd72085e65aecbd9e11f998d9e: Status 404 returned error can't find the container with id 818d760aeb79586c784eb67145d1ba318c410cbd72085e65aecbd9e11f998d9e
Jul 13 19:16:57.874259 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:57.874230    2581 generic.go:356] "Generic (PLEG): container finished" podID="796644df-e26a-4421-bbca-f793b5cad3fc" containerID="fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad" exitCode=0
Jul 13 19:16:57.874614 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:57.874276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" event={"ID":"796644df-e26a-4421-bbca-f793b5cad3fc","Type":"ContainerDied","Data":"fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad"}
Jul 13 19:16:57.874614 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:57.874311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" event={"ID":"796644df-e26a-4421-bbca-f793b5cad3fc","Type":"ContainerStarted","Data":"818d760aeb79586c784eb67145d1ba318c410cbd72085e65aecbd9e11f998d9e"}
Jul 13 19:16:57.904118 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:57.904086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:16:58.882169 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:58.882133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" event={"ID":"796644df-e26a-4421-bbca-f793b5cad3fc","Type":"ContainerStarted","Data":"9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c"}
Jul 13 19:16:58.882594 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:58.882241    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:16:58.901334 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:16:58.901286    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" podStartSLOduration=2.197871976 podStartE2EDuration="2.901268149s" totalImagesPullingTime="0s" totalInitContainerRuntime="703.396173ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:16:58.899240033 +0000 UTC m=+1655.799465427" watchObservedRunningTime="2026-07-13 19:16:58.901268149 +0000 UTC m=+1655.801493540"
Jul 13 19:17:05.174460 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:05.174429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:07.700868 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:07.700833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:09.760206 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:09.760141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"]
Jul 13 19:17:09.764377 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:09.764346    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:09.770379 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:17:09.770347    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:17:09.770379 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:17:09.770379 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:17:09.770379 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="a1c65586d1f844ffeaf49e23305009389be2abc1b0878d71d7e22643b4e44f08" cmd=["/bin/sleep","15"]
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:09.770392    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" message="" err=<
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:17:09.770403    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" podUID="11175152-4faf-48f5-a990-ca1782914956" containerName="main" containerID="cri-o://a1c65586d1f844ffeaf49e23305009389be2abc1b0878d71d7e22643b4e44f08"
Jul 13 19:17:09.770622 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:09.770442    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" podUID="11175152-4faf-48f5-a990-ca1782914956" containerName="main" containerID="cri-o://a1c65586d1f844ffeaf49e23305009389be2abc1b0878d71d7e22643b4e44f08" gracePeriod=60
Jul 13 19:17:09.826727 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:09.826687    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:09.938312 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:09.938284    2581 generic.go:356] "Generic (PLEG): container finished" podID="11175152-4faf-48f5-a990-ca1782914956" containerID="a1c65586d1f844ffeaf49e23305009389be2abc1b0878d71d7e22643b4e44f08" exitCode=0
Jul 13 19:17:09.938449 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:09.938331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" event={"ID":"11175152-4faf-48f5-a990-ca1782914956","Type":"ContainerDied","Data":"a1c65586d1f844ffeaf49e23305009389be2abc1b0878d71d7e22643b4e44f08"}
Jul 13 19:17:10.018311 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.018254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:17:10.171512 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.171480    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jddmv\" (UniqueName: \"kubernetes.io/projected/11175152-4faf-48f5-a990-ca1782914956-kube-api-access-jddmv\") pod \"11175152-4faf-48f5-a990-ca1782914956\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") "
Jul 13 19:17:10.171714 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.171538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11175152-4faf-48f5-a990-ca1782914956-tls-certs\") pod \"11175152-4faf-48f5-a990-ca1782914956\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") "
Jul 13 19:17:10.171714 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.171580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-tokenizer-uds\") pod \"11175152-4faf-48f5-a990-ca1782914956\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") "
Jul 13 19:17:10.171714 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.171638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-kserve-provision-location\") pod \"11175152-4faf-48f5-a990-ca1782914956\" (UID: \"11175152-4faf-48f5-a990-ca1782914956\") "
Jul 13 19:17:10.171936 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.171893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "11175152-4faf-48f5-a990-ca1782914956" (UID: "11175152-4faf-48f5-a990-ca1782914956"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:17:10.172351 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.172326    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11175152-4faf-48f5-a990-ca1782914956" (UID: "11175152-4faf-48f5-a990-ca1782914956"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:17:10.173836 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.173810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11175152-4faf-48f5-a990-ca1782914956-kube-api-access-jddmv" (OuterVolumeSpecName: "kube-api-access-jddmv") pod "11175152-4faf-48f5-a990-ca1782914956" (UID: "11175152-4faf-48f5-a990-ca1782914956"). InnerVolumeSpecName "kube-api-access-jddmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:17:10.173928 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.173852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11175152-4faf-48f5-a990-ca1782914956-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11175152-4faf-48f5-a990-ca1782914956" (UID: "11175152-4faf-48f5-a990-ca1782914956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:17:10.273282 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.273186    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:17:10.273282 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.273217    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddmv\" (UniqueName: \"kubernetes.io/projected/11175152-4faf-48f5-a990-ca1782914956-kube-api-access-jddmv\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:17:10.273282 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.273230    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11175152-4faf-48f5-a990-ca1782914956-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:17:10.273282 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.273242    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/11175152-4faf-48f5-a990-ca1782914956-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:17:10.944379 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.944349    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"
Jul 13 19:17:10.944379 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.944369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn" event={"ID":"11175152-4faf-48f5-a990-ca1782914956","Type":"ContainerDied","Data":"b1b980410ae87a07a90fb339b05b3facfef7f6bd156ba54c7182c218bb2545a8"}
Jul 13 19:17:10.944978 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.944436    2581 scope.go:117] "RemoveContainer" containerID="a1c65586d1f844ffeaf49e23305009389be2abc1b0878d71d7e22643b4e44f08"
Jul 13 19:17:10.953386 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.953371    2581 scope.go:117] "RemoveContainer" containerID="eab17495e6690e613e3df1e4adeaf024931a099c6f7a1be314fc06dccfc5b8e1"
Jul 13 19:17:10.964340 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.964316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"]
Jul 13 19:17:10.968426 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:10.968404    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn"]
Jul 13 19:17:11.603658 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:11.603628    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11175152-4faf-48f5-a990-ca1782914956" path="/var/lib/kubelet/pods/11175152-4faf-48f5-a990-ca1782914956/volumes"
Jul 13 19:17:29.865998 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:29.865961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:29.889296 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:29.889267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:17:29.966328 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:29.966296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:30.023997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:30.023966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:32.462377 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.462331    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:32.563489 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.563459    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:32.928868 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.928755    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"]
Jul 13 19:17:32.929112 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.929099    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11175152-4faf-48f5-a990-ca1782914956" containerName="storage-initializer"
Jul 13 19:17:32.929153 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.929115    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11175152-4faf-48f5-a990-ca1782914956" containerName="storage-initializer"
Jul 13 19:17:32.929153 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.929131    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11175152-4faf-48f5-a990-ca1782914956" containerName="main"
Jul 13 19:17:32.929153 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.929137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11175152-4faf-48f5-a990-ca1782914956" containerName="main"
Jul 13 19:17:32.929245 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.929186    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="11175152-4faf-48f5-a990-ca1782914956" containerName="main"
Jul 13 19:17:32.933224 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.933209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:32.937215 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:32.937187    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"]
Jul 13 19:17:33.045842 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:33.045781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:33.054399 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:33.054373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzj8\" (UniqueName: \"kubernetes.io/projected/a106df84-5ae6-429d-b8b1-9c8491c9264d-kube-api-access-7fzj8\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:33.155204 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:33.155173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzj8\" (UniqueName: \"kubernetes.io/projected/a106df84-5ae6-429d-b8b1-9c8491c9264d-kube-api-access-7fzj8\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:33.163499 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:33.163473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzj8\" (UniqueName: \"kubernetes.io/projected/a106df84-5ae6-429d-b8b1-9c8491c9264d-kube-api-access-7fzj8\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:38.596195 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:38.596153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:38.601445 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:38.601421    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:38.696712 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:38.696676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:38.696896 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:38.696782    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:38.699364 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:38.699343    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") DevicePath \"csi-db20c39a669aff37bf8e61ae40f04d3412fb83233df17a2be249c76baee257c5\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:39.109779 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:39.109731    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/20b3c94c5dbc323a50b49338bcc56e67a04735bee1ee50ef003b28c9268680c2/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:39.124182 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:39.124153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"e2e-pvc-model-download-w2mvs\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:39.243366 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:39.243330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:39.377308 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:39.377283    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"]
Jul 13 19:17:39.379378 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:17:39.379348    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda106df84_5ae6_429d_b8b1_9c8491c9264d.slice/crio-29a0a06000fcad82424429ba831e040fb39cc060dc8edde2df5e9007689ac950 WatchSource:0}: Error finding container 29a0a06000fcad82424429ba831e040fb39cc060dc8edde2df5e9007689ac950: Status 404 returned error can't find the container with id 29a0a06000fcad82424429ba831e040fb39cc060dc8edde2df5e9007689ac950
Jul 13 19:17:40.079442 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:40.079397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs" event={"ID":"a106df84-5ae6-429d-b8b1-9c8491c9264d","Type":"ContainerStarted","Data":"b7273bf6d4c67722c8dda38519399b8df1667a1aa780451140a9988708c4ec7a"}
Jul 13 19:17:40.079860 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:40.079457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs" event={"ID":"a106df84-5ae6-429d-b8b1-9c8491c9264d","Type":"ContainerStarted","Data":"29a0a06000fcad82424429ba831e040fb39cc060dc8edde2df5e9007689ac950"}
Jul 13 19:17:40.096657 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:40.096591    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs" podStartSLOduration=11.096568891 podStartE2EDuration="11.096568891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:17:40.092982679 +0000 UTC m=+1696.993208072" watchObservedRunningTime="2026-07-13 19:17:40.096568891 +0000 UTC m=+1696.996794284"
Jul 13 19:17:46.112510 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:46.112410    2581 generic.go:356] "Generic (PLEG): container finished" podID="a106df84-5ae6-429d-b8b1-9c8491c9264d" containerID="b7273bf6d4c67722c8dda38519399b8df1667a1aa780451140a9988708c4ec7a" exitCode=0
Jul 13 19:17:46.112510 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:46.112488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs" event={"ID":"a106df84-5ae6-429d-b8b1-9c8491c9264d","Type":"ContainerDied","Data":"b7273bf6d4c67722c8dda38519399b8df1667a1aa780451140a9988708c4ec7a"}
Jul 13 19:17:47.269248 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.269225    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:47.367829 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.367701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fzj8\" (UniqueName: \"kubernetes.io/projected/a106df84-5ae6-429d-b8b1-9c8491c9264d-kube-api-access-7fzj8\") pod \"a106df84-5ae6-429d-b8b1-9c8491c9264d\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") "
Jul 13 19:17:47.367988 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.367845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") pod \"a106df84-5ae6-429d-b8b1-9c8491c9264d\" (UID: \"a106df84-5ae6-429d-b8b1-9c8491c9264d\") "
Jul 13 19:17:47.370002 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.369971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a106df84-5ae6-429d-b8b1-9c8491c9264d-kube-api-access-7fzj8" (OuterVolumeSpecName: "kube-api-access-7fzj8") pod "a106df84-5ae6-429d-b8b1-9c8491c9264d" (UID: "a106df84-5ae6-429d-b8b1-9c8491c9264d"). InnerVolumeSpecName "kube-api-access-7fzj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:17:47.371678 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.371656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b" (OuterVolumeSpecName: "model-storage") pod "a106df84-5ae6-429d-b8b1-9c8491c9264d" (UID: "a106df84-5ae6-429d-b8b1-9c8491c9264d"). InnerVolumeSpecName "pvc-3823dadc-7c52-412c-8ef9-84caab00a904". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 13 19:17:47.469020 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.468984    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fzj8\" (UniqueName: \"kubernetes.io/projected/a106df84-5ae6-429d-b8b1-9c8491c9264d-kube-api-access-7fzj8\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:17:47.469179 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.469036    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") on node \"ip-10-0-131-203.ec2.internal\" "
Jul 13 19:17:47.575085 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.575065    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b") on node "ip-10-0-131-203.ec2.internal"
Jul 13 19:17:47.670848 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:47.670747    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-3823dadc-7c52-412c-8ef9-84caab00a904\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0332b6a88ccce378b\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"csi-db20c39a669aff37bf8e61ae40f04d3412fb83233df17a2be249c76baee257c5\""
Jul 13 19:17:48.123291 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:48.123264    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"
Jul 13 19:17:48.123291 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:48.123287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs" event={"ID":"a106df84-5ae6-429d-b8b1-9c8491c9264d","Type":"ContainerDied","Data":"29a0a06000fcad82424429ba831e040fb39cc060dc8edde2df5e9007689ac950"}
Jul 13 19:17:48.123530 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:48.123319    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a0a06000fcad82424429ba831e040fb39cc060dc8edde2df5e9007689ac950"
Jul 13 19:17:51.840772 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:51.840741    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:17:52.055724 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:17:52.055690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:18:05.606023 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:18:05.605987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:19:07.102660 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.102626    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"]
Jul 13 19:19:07.111165 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.111137    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:19:07.112746 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:19:07.112710    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:19:07.112746 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:19:07.112746 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:19:07.112746 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c" cmd=["/bin/sleep","15"]
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.112763    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" message="" err=<
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:19:07.112777    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" containerName="main" containerID="cri-o://9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c"
Jul 13 19:19:07.113056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.112825    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" containerName="main" containerID="cri-o://9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c" gracePeriod=60
Jul 13 19:19:07.168598 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.168563    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:19:07.360981 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.360906    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:19:07.385044 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.385007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-tokenizer-uds\") pod \"796644df-e26a-4421-bbca-f793b5cad3fc\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") "
Jul 13 19:19:07.385183 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.385049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5fwg\" (UniqueName: \"kubernetes.io/projected/796644df-e26a-4421-bbca-f793b5cad3fc-kube-api-access-p5fwg\") pod \"796644df-e26a-4421-bbca-f793b5cad3fc\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") "
Jul 13 19:19:07.385183 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.385069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-kserve-provision-location\") pod \"796644df-e26a-4421-bbca-f793b5cad3fc\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") "
Jul 13 19:19:07.385183 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.385092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796644df-e26a-4421-bbca-f793b5cad3fc-tls-certs\") pod \"796644df-e26a-4421-bbca-f793b5cad3fc\" (UID: \"796644df-e26a-4421-bbca-f793b5cad3fc\") "
Jul 13 19:19:07.385341 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.385246    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "796644df-e26a-4421-bbca-f793b5cad3fc" (UID: "796644df-e26a-4421-bbca-f793b5cad3fc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:19:07.385341 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.385304    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:19:07.386019 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.385772    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "796644df-e26a-4421-bbca-f793b5cad3fc" (UID: "796644df-e26a-4421-bbca-f793b5cad3fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:19:07.387380 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.387345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796644df-e26a-4421-bbca-f793b5cad3fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "796644df-e26a-4421-bbca-f793b5cad3fc" (UID: "796644df-e26a-4421-bbca-f793b5cad3fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:19:07.387506 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.387398    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796644df-e26a-4421-bbca-f793b5cad3fc-kube-api-access-p5fwg" (OuterVolumeSpecName: "kube-api-access-p5fwg") pod "796644df-e26a-4421-bbca-f793b5cad3fc" (UID: "796644df-e26a-4421-bbca-f793b5cad3fc"). InnerVolumeSpecName "kube-api-access-p5fwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:19:07.486170 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.486136    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5fwg\" (UniqueName: \"kubernetes.io/projected/796644df-e26a-4421-bbca-f793b5cad3fc-kube-api-access-p5fwg\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:19:07.486170 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.486161    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/796644df-e26a-4421-bbca-f793b5cad3fc-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:19:07.486170 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.486172    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796644df-e26a-4421-bbca-f793b5cad3fc-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:19:07.503843 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.503800    2581 generic.go:356] "Generic (PLEG): container finished" podID="796644df-e26a-4421-bbca-f793b5cad3fc" containerID="9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c" exitCode=0
Jul 13 19:19:07.503979 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.503861    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"
Jul 13 19:19:07.503979 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.503884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" event={"ID":"796644df-e26a-4421-bbca-f793b5cad3fc","Type":"ContainerDied","Data":"9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c"}
Jul 13 19:19:07.503979 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.503955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5" event={"ID":"796644df-e26a-4421-bbca-f793b5cad3fc","Type":"ContainerDied","Data":"818d760aeb79586c784eb67145d1ba318c410cbd72085e65aecbd9e11f998d9e"}
Jul 13 19:19:07.503979 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.503976    2581 scope.go:117] "RemoveContainer" containerID="9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c"
Jul 13 19:19:07.515323 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.515306    2581 scope.go:117] "RemoveContainer" containerID="fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad"
Jul 13 19:19:07.523211 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.523193    2581 scope.go:117] "RemoveContainer" containerID="9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c"
Jul 13 19:19:07.523467 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:19:07.523448    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c\": container with ID starting with 9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c not found: ID does not exist" containerID="9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c"
Jul 13 19:19:07.523515 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.523475    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c"} err="failed to get container status \"9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c\": rpc error: code = NotFound desc = could not find container \"9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c\": container with ID starting with 9a6cabc02214d2175b87ac318d4511f44bebdc4f97b9e9e5b338dd9e7aba9e8c not found: ID does not exist"
Jul 13 19:19:07.523515 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.523491    2581 scope.go:117] "RemoveContainer" containerID="fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad"
Jul 13 19:19:07.523719 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:19:07.523700    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad\": container with ID starting with fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad not found: ID does not exist" containerID="fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad"
Jul 13 19:19:07.523759 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.523726    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad"} err="failed to get container status \"fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad\": rpc error: code = NotFound desc = could not find container \"fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad\": container with ID starting with fbacbc050c5f57d1685d398b5042f6f160fbd012532a95227a27e1c71933b4ad not found: ID does not exist"
Jul 13 19:19:07.549151 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.549128    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"]
Jul 13 19:19:07.561970 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.561951    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5"]
Jul 13 19:19:07.602855 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:07.602829    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" path="/var/lib/kubelet/pods/796644df-e26a-4421-bbca-f793b5cad3fc/volumes"
Jul 13 19:19:23.662355 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:23.662328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:19:23.665697 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:23.665673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:19:31.697989 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.697957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:19:31.915421 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915380    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"]
Jul 13 19:19:31.915693 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915679    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" containerName="storage-initializer"
Jul 13 19:19:31.915693 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915694    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" containerName="storage-initializer"
Jul 13 19:19:31.915832 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915705    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a106df84-5ae6-429d-b8b1-9c8491c9264d" containerName="storage-initializer"
Jul 13 19:19:31.915832 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915710    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a106df84-5ae6-429d-b8b1-9c8491c9264d" containerName="storage-initializer"
Jul 13 19:19:31.915832 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915722    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" containerName="main"
Jul 13 19:19:31.915832 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915728    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" containerName="main"
Jul 13 19:19:31.915832 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915776    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a106df84-5ae6-429d-b8b1-9c8491c9264d" containerName="storage-initializer"
Jul 13 19:19:31.915832 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.915804    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="796644df-e26a-4421-bbca-f793b5cad3fc" containerName="main"
Jul 13 19:19:31.918354 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.918334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:31.920856 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.920827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:19:31.921175 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.921159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:19:31.921652 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.921626    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-8hsbh\""
Jul 13 19:19:31.921739 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.921730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:19:31.921868 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.921850    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 13 19:19:31.922017 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.922000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:19:31.931156 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.931131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"]
Jul 13 19:19:31.951564 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.951492    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:19:31.965647 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.965621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:31.965738 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.965660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2qd\" (UniqueName: \"kubernetes.io/projected/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kube-api-access-mp2qd\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:31.965783 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.965755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:31.965783 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:31.965784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.066292 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.066251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.066292 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.066295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.066528 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.066331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.066528 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.066375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2qd\" (UniqueName: \"kubernetes.io/projected/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kube-api-access-mp2qd\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.066646 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.066609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.066741 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.066647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.069095 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.069069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.075224 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.075198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2qd\" (UniqueName: \"kubernetes.io/projected/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kube-api-access-mp2qd\") pod \"custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.227401 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.227369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:32.359262 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.359237    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"]
Jul 13 19:19:32.361310 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:19:32.361271    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd64d12c_39bc_4519_bed7_1c7e8b497e3d.slice/crio-8c760475c83abebb08285c93ff157e389940184d80bc0bf16ea2011a4a2cb309 WatchSource:0}: Error finding container 8c760475c83abebb08285c93ff157e389940184d80bc0bf16ea2011a4a2cb309: Status 404 returned error can't find the container with id 8c760475c83abebb08285c93ff157e389940184d80bc0bf16ea2011a4a2cb309
Jul 13 19:19:32.626010 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.625919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" event={"ID":"bd64d12c-39bc-4519-bed7-1c7e8b497e3d","Type":"ContainerStarted","Data":"efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1"}
Jul 13 19:19:32.626010 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:32.625979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" event={"ID":"bd64d12c-39bc-4519-bed7-1c7e8b497e3d","Type":"ContainerStarted","Data":"8c760475c83abebb08285c93ff157e389940184d80bc0bf16ea2011a4a2cb309"}
Jul 13 19:19:33.633378 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:33.633340    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerID="efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1" exitCode=0
Jul 13 19:19:33.633806 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:33.633391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" event={"ID":"bd64d12c-39bc-4519-bed7-1c7e8b497e3d","Type":"ContainerDied","Data":"efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1"}
Jul 13 19:19:34.640133 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:34.640100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" event={"ID":"bd64d12c-39bc-4519-bed7-1c7e8b497e3d","Type":"ContainerStarted","Data":"029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca"}
Jul 13 19:19:34.640512 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:34.640164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:19:34.659068 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:19:34.659013    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" podStartSLOduration=2.894642848 podStartE2EDuration="3.658997043s" totalImagesPullingTime="0s" totalInitContainerRuntime="764.354195ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:19:34.658394411 +0000 UTC m=+1811.558619815" watchObservedRunningTime="2026-07-13 19:19:34.658997043 +0000 UTC m=+1811.559222455"
Jul 13 19:20:05.647132 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:20:05.647102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:20:26.401008 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:20:26.400974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:21:45.236185 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.236145    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"]
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:21:45.245703    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca" cmd=["/bin/sleep","15"]
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.245750    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" message="" err=<
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:21:45.245736 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:21:45.246301 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:21:45.245761    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:21:45.246301 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:21:45.246301 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:21:45.246301 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerName="main" containerID="cri-o://029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca"
Jul 13 19:21:45.246301 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.245819    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerName="main" containerID="cri-o://029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca" gracePeriod=60
Jul 13 19:21:45.249383 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.249296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:21:45.336174 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.336139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:21:45.492117 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.492047    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:21:45.597097 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.597065    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kserve-provision-location\") pod \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") "
Jul 13 19:21:45.597255 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.597110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tokenizer-uds\") pod \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") "
Jul 13 19:21:45.597255 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.597141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp2qd\" (UniqueName: \"kubernetes.io/projected/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kube-api-access-mp2qd\") pod \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") "
Jul 13 19:21:45.597255 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.597175    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tls-certs\") pod \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\" (UID: \"bd64d12c-39bc-4519-bed7-1c7e8b497e3d\") "
Jul 13 19:21:45.597440 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.597403    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bd64d12c-39bc-4519-bed7-1c7e8b497e3d" (UID: "bd64d12c-39bc-4519-bed7-1c7e8b497e3d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:21:45.597894 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.597809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd64d12c-39bc-4519-bed7-1c7e8b497e3d" (UID: "bd64d12c-39bc-4519-bed7-1c7e8b497e3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:21:45.599427 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.599400    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd64d12c-39bc-4519-bed7-1c7e8b497e3d" (UID: "bd64d12c-39bc-4519-bed7-1c7e8b497e3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:21:45.599582 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.599499    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kube-api-access-mp2qd" (OuterVolumeSpecName: "kube-api-access-mp2qd") pod "bd64d12c-39bc-4519-bed7-1c7e8b497e3d" (UID: "bd64d12c-39bc-4519-bed7-1c7e8b497e3d"). InnerVolumeSpecName "kube-api-access-mp2qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:21:45.697944 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.697913    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:21:45.697944 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.697942    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:21:45.697944 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.697952    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2qd\" (UniqueName: \"kubernetes.io/projected/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-kube-api-access-mp2qd\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:21:45.698211 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:45.697961    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd64d12c-39bc-4519-bed7-1c7e8b497e3d-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:21:46.273429 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.273395    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerID="029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca" exitCode=0
Jul 13 19:21:46.273931 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.273461    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"
Jul 13 19:21:46.273931 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.273461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" event={"ID":"bd64d12c-39bc-4519-bed7-1c7e8b497e3d","Type":"ContainerDied","Data":"029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca"}
Jul 13 19:21:46.273931 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.273531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6" event={"ID":"bd64d12c-39bc-4519-bed7-1c7e8b497e3d","Type":"ContainerDied","Data":"8c760475c83abebb08285c93ff157e389940184d80bc0bf16ea2011a4a2cb309"}
Jul 13 19:21:46.273931 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.273561    2581 scope.go:117] "RemoveContainer" containerID="029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca"
Jul 13 19:21:46.282442 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.282427    2581 scope.go:117] "RemoveContainer" containerID="efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1"
Jul 13 19:21:46.290349 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.290332    2581 scope.go:117] "RemoveContainer" containerID="029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca"
Jul 13 19:21:46.290617 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:21:46.290598    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca\": container with ID starting with 029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca not found: ID does not exist" containerID="029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca"
Jul 13 19:21:46.290667 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.290627    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca"} err="failed to get container status \"029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca\": rpc error: code = NotFound desc = could not find container \"029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca\": container with ID starting with 029a833120cfd377a043b604630d87e4717a95a02d49e43b679e111e405fe5ca not found: ID does not exist"
Jul 13 19:21:46.290667 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.290644    2581 scope.go:117] "RemoveContainer" containerID="efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1"
Jul 13 19:21:46.290937 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:21:46.290919    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1\": container with ID starting with efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1 not found: ID does not exist" containerID="efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1"
Jul 13 19:21:46.290996 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.290948    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1"} err="failed to get container status \"efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1\": rpc error: code = NotFound desc = could not find container \"efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1\": container with ID starting with efe8619eac6b82eda07271e4418a3ddb1b25c52e3bf1751e3eb6cfd6a0f149f1 not found: ID does not exist"
Jul 13 19:21:46.313850 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.313814    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"]
Jul 13 19:21:46.325039 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:46.325010    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6"]
Jul 13 19:21:47.607495 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:21:47.607463    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" path="/var/lib/kubelet/pods/bd64d12c-39bc-4519-bed7-1c7e8b497e3d/volumes"
Jul 13 19:22:06.293038 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.292996    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"]
Jul 13 19:22:06.293402 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.293340    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerName="storage-initializer"
Jul 13 19:22:06.293402 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.293353    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerName="storage-initializer"
Jul 13 19:22:06.293402 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.293360    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerName="main"
Jul 13 19:22:06.293402 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.293365    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerName="main"
Jul 13 19:22:06.293535 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.293422    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd64d12c-39bc-4519-bed7-1c7e8b497e3d" containerName="main"
Jul 13 19:22:06.296297 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.296272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.298524 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.298505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zh96m\""
Jul 13 19:22:06.298918 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.298897    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:22:06.299035 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.298944    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 13 19:22:06.300132 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.300105    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:22:06.315744 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.315716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"]
Jul 13 19:22:06.355973 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.355934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356135 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.355980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356135 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.356034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356135 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.356058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356261 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.356129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356261 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.356169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-kube-api-access-fr7fg\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356261 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.356198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356261 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.356233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.356402 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.356281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.457646 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.457861 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.457861 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.457861 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.457861 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.457861 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458152 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458152 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-kube-api-access-fr7fg\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458152 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.457936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458152 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.458126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458446 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.458376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458738 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.458453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458738 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.458461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.458946 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.458904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.460286 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.460261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.460575 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.460557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.468102 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.468073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-kube-api-access-fr7fg\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.468207 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.468148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-rbgv4\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.607328 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.607244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:06.750634 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.750593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"]
Jul 13 19:22:06.753667 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:22:06.753637    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bc7aeb_3257_4e2a_b0b9_48f3ecf40f41.slice/crio-9b4e85c0e28a65ca4bb9a88e1971c0ba8c34d461b00b1fd6c0b8ed63675ed106 WatchSource:0}: Error finding container 9b4e85c0e28a65ca4bb9a88e1971c0ba8c34d461b00b1fd6c0b8ed63675ed106: Status 404 returned error can't find the container with id 9b4e85c0e28a65ca4bb9a88e1971c0ba8c34d461b00b1fd6c0b8ed63675ed106
Jul 13 19:22:06.755673 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.755639    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:06.755781 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.755729    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:06.755781 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:06.755772    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:07.373987 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:07.373948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" event={"ID":"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41","Type":"ContainerStarted","Data":"51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda"}
Jul 13 19:22:07.374387 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:07.374009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" event={"ID":"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41","Type":"ContainerStarted","Data":"9b4e85c0e28a65ca4bb9a88e1971c0ba8c34d461b00b1fd6c0b8ed63675ed106"}
Jul 13 19:22:07.411438 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:07.411382    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podStartSLOduration=1.411366111 podStartE2EDuration="1.411366111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:07.410864784 +0000 UTC m=+1964.311090177" watchObservedRunningTime="2026-07-13 19:22:07.411366111 +0000 UTC m=+1964.311591480"
Jul 13 19:22:07.607411 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:07.607380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:07.613175 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:07.613149    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:08.378617 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:08.378578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:08.379583 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:08.379563    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:09.016289 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.016250    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"]
Jul 13 19:22:09.019268 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.019252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.038431 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.038404    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"]
Jul 13 19:22:09.079584 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlm8\" (UniqueName: \"kubernetes.io/projected/9f1416fa-ab83-4554-ab21-2f2312113b39-kube-api-access-7vlm8\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1416fa-ab83-4554-ab21-2f2312113b39-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079989 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079989 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.079989 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.079865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.180704 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.180923 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.180923 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.180923 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.180923 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.180923 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.180923 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.181238 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlm8\" (UniqueName: \"kubernetes.io/projected/9f1416fa-ab83-4554-ab21-2f2312113b39-kube-api-access-7vlm8\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.181238 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.180971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1416fa-ab83-4554-ab21-2f2312113b39-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.181238 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.181175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.181238 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.181218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.181238 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.181234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.181471 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.181369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.181691 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.181670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1416fa-ab83-4554-ab21-2f2312113b39-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.183460 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.183442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.183540 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.183497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.188942 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.188917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1416fa-ab83-4554-ab21-2f2312113b39-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.189071 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.189052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlm8\" (UniqueName: \"kubernetes.io/projected/9f1416fa-ab83-4554-ab21-2f2312113b39-kube-api-access-7vlm8\") pod \"router-gateway-1-openshift-default-54c47cdfc7-r2bjb\" (UID: \"9f1416fa-ab83-4554-ab21-2f2312113b39\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.329760 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.329678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:09.476401 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.476373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"]
Jul 13 19:22:09.479170 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:22:09.479144    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1416fa_ab83_4554_ab21_2f2312113b39.slice/crio-54462a828a946cc8f16e1d145759b9f703145a757221426311e31fb72f43f21c WatchSource:0}: Error finding container 54462a828a946cc8f16e1d145759b9f703145a757221426311e31fb72f43f21c: Status 404 returned error can't find the container with id 54462a828a946cc8f16e1d145759b9f703145a757221426311e31fb72f43f21c
Jul 13 19:22:09.481584 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.481554    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:09.481681 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.481632    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:09.481744 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:09.481677    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:10.390497 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:10.390455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" event={"ID":"9f1416fa-ab83-4554-ab21-2f2312113b39","Type":"ContainerStarted","Data":"4c8f27437b3de9d73eb5c37ddaadbb835cd5ae82a3db90c05b3b0baecf3a772c"}
Jul 13 19:22:10.390685 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:10.390517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" event={"ID":"9f1416fa-ab83-4554-ab21-2f2312113b39","Type":"ContainerStarted","Data":"54462a828a946cc8f16e1d145759b9f703145a757221426311e31fb72f43f21c"}
Jul 13 19:22:10.414000 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:10.413947    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podStartSLOduration=2.4139300009999998 podStartE2EDuration="2.413930001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:10.411622235 +0000 UTC m=+1967.311847653" watchObservedRunningTime="2026-07-13 19:22:10.413930001 +0000 UTC m=+1967.314155393"
Jul 13 19:22:11.331986 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:11.331951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:12.331183 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:12.331124    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:13.336060 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:13.336001    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:14.378026 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:14.377977    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:15.390145 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:15.390086    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:16.391645 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:16.391594    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:17.433993 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:17.433946    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:18.475983 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:18.475930    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:19.451987 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.451898    2581 generic.go:356] "Generic (PLEG): container finished" podID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerID="51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda" exitCode=0
Jul 13 19:22:19.452207 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.451982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" event={"ID":"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41","Type":"ContainerDied","Data":"51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda"}
Jul 13 19:22:19.452434 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.452418    2581 scope.go:117] "RemoveContainer" containerID="51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda"
Jul 13 19:22:19.453506 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.453473    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:19.453603 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.453558    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:19.453644 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.453606    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:19.516960 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.516922    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 13 19:22:19.795679 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:19.795640    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": read tcp 10.132.0.2:35418->10.132.0.46:15021: read: connection reset by peer"
Jul 13 19:22:20.330836 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.330770    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 13 19:22:20.458461 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.458427    2581 generic.go:356] "Generic (PLEG): container finished" podID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerID="4c8f27437b3de9d73eb5c37ddaadbb835cd5ae82a3db90c05b3b0baecf3a772c" exitCode=0
Jul 13 19:22:20.458649 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.458502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" event={"ID":"9f1416fa-ab83-4554-ab21-2f2312113b39","Type":"ContainerDied","Data":"4c8f27437b3de9d73eb5c37ddaadbb835cd5ae82a3db90c05b3b0baecf3a772c"}
Jul 13 19:22:20.458954 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.458936    2581 scope.go:117] "RemoveContainer" containerID="4c8f27437b3de9d73eb5c37ddaadbb835cd5ae82a3db90c05b3b0baecf3a772c"
Jul 13 19:22:20.459963 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.459934    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:20.460053 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.460012    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:20.460117 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.460049    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:22:20.460641 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.460620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" event={"ID":"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41","Type":"ContainerStarted","Data":"5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8"}
Jul 13 19:22:20.608541 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.608399    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:20.609299 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:20.609262    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 13 19:22:21.330111 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:21.330045    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:21.467918 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:21.467888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" event={"ID":"9f1416fa-ab83-4554-ab21-2f2312113b39","Type":"ContainerStarted","Data":"ca52a006b3b03d2cf6d195408a4773d06b0155e3ed039535a0bf1d96c8d94601"}
Jul 13 19:22:21.607758 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:21.607658    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 13 19:22:22.331158 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:22.331115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:22.331683 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:22.331246    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 13 19:22:22.608417 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:22.608289    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 13 19:22:23.330325 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:23.330278    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 13 19:22:23.378703 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:23.378664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:23.608281 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:23.608185    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 13 19:22:24.330250 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:24.330166    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:24.330512 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:24.330472    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 13 19:22:24.607949 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:24.607836    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 13 19:22:25.330868 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:25.330818    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 13 19:22:25.608652 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:25.608548    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 13 19:22:25.774736 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:25.774696    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:26.294812 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.294325    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"]
Jul 13 19:22:26.309600 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.309070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.313489 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.313457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:22:26.314049 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.314028    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:22:26.314184 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.314124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 13 19:22:26.322717 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.322692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"]
Jul 13 19:22:26.324987 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.324966    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-m9d98\""
Jul 13 19:22:26.330292 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.330262    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb" podUID="9f1416fa-ab83-4554-ab21-2f2312113b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 13 19:22:26.457098 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.457064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.457295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.457120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.457295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.457152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.457295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.457170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cm5g\" (UniqueName: \"kubernetes.io/projected/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kube-api-access-4cm5g\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.558655 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.558561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.558655 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.558616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.558655 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.558635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cm5g\" (UniqueName: \"kubernetes.io/projected/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kube-api-access-4cm5g\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.558921 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.558680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.559067 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.559042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.559164 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.559137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.561458 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.561437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.568200 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.568170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cm5g\" (UniqueName: \"kubernetes.io/projected/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kube-api-access-4cm5g\") pod \"router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.619954 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.619911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:26.637401 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.637358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:26.638570 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.638545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:26.789560 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:26.789533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"]
Jul 13 19:22:26.791733 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:22:26.791684    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684d5b95_817e_4ec8_ac00_7dbd198f07c3.slice/crio-c460e7068315d9e2f3ed29eeb6c869630126ad90326446222382d6ace09b8545 WatchSource:0}: Error finding container c460e7068315d9e2f3ed29eeb6c869630126ad90326446222382d6ace09b8545: Status 404 returned error can't find the container with id c460e7068315d9e2f3ed29eeb6c869630126ad90326446222382d6ace09b8545
Jul 13 19:22:27.333730 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:27.333701    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:27.334606 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:27.334583    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb"
Jul 13 19:22:27.403678 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:27.403645    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"]
Jul 13 19:22:27.504897 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:27.504861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" event={"ID":"684d5b95-817e-4ec8-ac00-7dbd198f07c3","Type":"ContainerStarted","Data":"6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a"}
Jul 13 19:22:27.505049 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:27.504912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" event={"ID":"684d5b95-817e-4ec8-ac00-7dbd198f07c3","Type":"ContainerStarted","Data":"c460e7068315d9e2f3ed29eeb6c869630126ad90326446222382d6ace09b8545"}
Jul 13 19:22:27.770939 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:27.770907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:22:28.511629 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:28.511593    2581 generic.go:356] "Generic (PLEG): container finished" podID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerID="6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a" exitCode=0
Jul 13 19:22:28.511845 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:28.511665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" event={"ID":"684d5b95-817e-4ec8-ac00-7dbd198f07c3","Type":"ContainerDied","Data":"6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a"}
Jul 13 19:22:28.511927 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:28.511880    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy" containerID="cri-o://5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8" gracePeriod=30
Jul 13 19:22:29.518206 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:29.518172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" event={"ID":"684d5b95-817e-4ec8-ac00-7dbd198f07c3","Type":"ContainerStarted","Data":"f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6"}
Jul 13 19:22:29.518609 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:29.518259    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:22:29.536580 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:29.536523    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" podStartSLOduration=2.793838954 podStartE2EDuration="3.536506813s" totalImagesPullingTime="0s" totalInitContainerRuntime="742.667859ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:29.535964618 +0000 UTC m=+1986.436190021" watchObservedRunningTime="2026-07-13 19:22:29.536506813 +0000 UTC m=+1986.436732232"
Jul 13 19:22:33.776522 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.776463    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:33.923602 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-certs\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.923764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-credential-socket\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.923764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-socket\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.923764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-kube-api-access-fr7fg\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.923764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-data\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.923764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923723    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-envoy\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.923764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-podinfo\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.924220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923777    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-token\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.924220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istiod-ca-cert\") pod \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\" (UID: \"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41\") "
Jul 13 19:22:33.924220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.923933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:33.924220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.924041    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:33.924220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.924052    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:33.924220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.924094    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:33.924220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.924127    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-data" (OuterVolumeSpecName: "istio-data") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:33.924532 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.924446    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:22:33.926135 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.926102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 19:22:33.926243 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.926167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:33.926243 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.926181    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-kube-api-access-fr7fg" (OuterVolumeSpecName: "kube-api-access-fr7fg") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "kube-api-access-fr7fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:33.926326 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:33.926243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-token" (OuterVolumeSpecName: "istio-token") pod "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" (UID: "08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:34.025288 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025238    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-envoy\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.025288 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025284    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-podinfo\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.025288 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025293    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-token\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.025288 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025301    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istiod-ca-cert\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.025586 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025310    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-credential-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.025586 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025318    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-workload-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.025586 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025329    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr7fg\" (UniqueName: \"kubernetes.io/projected/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-kube-api-access-fr7fg\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.025586 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.025337    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41-istio-data\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:22:34.544309 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.544277    2581 generic.go:356] "Generic (PLEG): container finished" podID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerID="5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8" exitCode=0
Jul 13 19:22:34.544485 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.544333    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"
Jul 13 19:22:34.544485 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.544364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" event={"ID":"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41","Type":"ContainerDied","Data":"5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8"}
Jul 13 19:22:34.544485 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.544451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4" event={"ID":"08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41","Type":"ContainerDied","Data":"9b4e85c0e28a65ca4bb9a88e1971c0ba8c34d461b00b1fd6c0b8ed63675ed106"}
Jul 13 19:22:34.544485 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.544481    2581 scope.go:117] "RemoveContainer" containerID="5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8"
Jul 13 19:22:34.556094 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.556072    2581 scope.go:117] "RemoveContainer" containerID="51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda"
Jul 13 19:22:34.564088 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.564069    2581 scope.go:117] "RemoveContainer" containerID="5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8"
Jul 13 19:22:34.564333 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:22:34.564313    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8\": container with ID starting with 5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8 not found: ID does not exist" containerID="5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8"
Jul 13 19:22:34.564396 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.564344    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8"} err="failed to get container status \"5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8\": rpc error: code = NotFound desc = could not find container \"5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8\": container with ID starting with 5d9c4a92c825c41ce4079d1a80fe32b843f2380d6e6b7abb65d6ab1cd6e48ec8 not found: ID does not exist"
Jul 13 19:22:34.564396 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.564367    2581 scope.go:117] "RemoveContainer" containerID="51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda"
Jul 13 19:22:34.564731 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:22:34.564675    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda\": container with ID starting with 51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda not found: ID does not exist" containerID="51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda"
Jul 13 19:22:34.564731 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.564740    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda"} err="failed to get container status \"51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda\": rpc error: code = NotFound desc = could not find container \"51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda\": container with ID starting with 51c3407e8dfad1b653ffa41b45cc83187296f1feee49e34a52f35491d0e01fda not found: ID does not exist"
Jul 13 19:22:34.567033 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.567014    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"]
Jul 13 19:22:34.571146 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:34.571125    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4"]
Jul 13 19:22:35.603599 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:22:35.603569    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" path="/var/lib/kubelet/pods/08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41/volumes"
Jul 13 19:23:00.524704 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:23:00.524669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:23:51.549277 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:23:51.549197    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:24:23.691367 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:24:23.691340    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:24:23.697348 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:24:23.697328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:25:31.459917 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.459827    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"]
Jul 13 19:25:31.468950 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:25:31.468888    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:25:31.468950 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:25:31.468950 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:25:31.468950 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6" cmd=["/bin/sleep","15"]
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.468952    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" message="" err=<
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:25:31.468968    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerName="main" containerID="cri-o://f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6"
Jul 13 19:25:31.469233 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.469011    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerName="main" containerID="cri-o://f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6" gracePeriod=60
Jul 13 19:25:31.488741 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.488706    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:25:31.582286 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.582232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:25:31.722577 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.722521    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:25:31.872941 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.872906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tls-certs\") pod \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") "
Jul 13 19:25:31.873115 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.872965    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cm5g\" (UniqueName: \"kubernetes.io/projected/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kube-api-access-4cm5g\") pod \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") "
Jul 13 19:25:31.873115 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.872990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tokenizer-uds\") pod \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") "
Jul 13 19:25:31.873115 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.873020    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kserve-provision-location\") pod \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\" (UID: \"684d5b95-817e-4ec8-ac00-7dbd198f07c3\") "
Jul 13 19:25:31.873314 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.873276    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "684d5b95-817e-4ec8-ac00-7dbd198f07c3" (UID: "684d5b95-817e-4ec8-ac00-7dbd198f07c3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:25:31.873751 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.873727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "684d5b95-817e-4ec8-ac00-7dbd198f07c3" (UID: "684d5b95-817e-4ec8-ac00-7dbd198f07c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:25:31.875226 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.875199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kube-api-access-4cm5g" (OuterVolumeSpecName: "kube-api-access-4cm5g") pod "684d5b95-817e-4ec8-ac00-7dbd198f07c3" (UID: "684d5b95-817e-4ec8-ac00-7dbd198f07c3"). InnerVolumeSpecName "kube-api-access-4cm5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:25:31.875339 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.875242    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "684d5b95-817e-4ec8-ac00-7dbd198f07c3" (UID: "684d5b95-817e-4ec8-ac00-7dbd198f07c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:25:31.974051 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.973973    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cm5g\" (UniqueName: \"kubernetes.io/projected/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kube-api-access-4cm5g\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:25:31.974051 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.974000    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:25:31.974051 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.974010    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684d5b95-817e-4ec8-ac00-7dbd198f07c3-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:25:31.974051 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:31.974020    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d5b95-817e-4ec8-ac00-7dbd198f07c3-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:25:32.324695 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.324589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:25:32.397167 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.397125    2581 generic.go:356] "Generic (PLEG): container finished" podID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerID="f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6" exitCode=0
Jul 13 19:25:32.397369 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.397182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" event={"ID":"684d5b95-817e-4ec8-ac00-7dbd198f07c3","Type":"ContainerDied","Data":"f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6"}
Jul 13 19:25:32.397369 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.397186    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"
Jul 13 19:25:32.397369 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.397222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf" event={"ID":"684d5b95-817e-4ec8-ac00-7dbd198f07c3","Type":"ContainerDied","Data":"c460e7068315d9e2f3ed29eeb6c869630126ad90326446222382d6ace09b8545"}
Jul 13 19:25:32.397369 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.397241    2581 scope.go:117] "RemoveContainer" containerID="f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6"
Jul 13 19:25:32.407157 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.407141    2581 scope.go:117] "RemoveContainer" containerID="6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a"
Jul 13 19:25:32.414962 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.414945    2581 scope.go:117] "RemoveContainer" containerID="f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6"
Jul 13 19:25:32.415192 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:25:32.415164    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6\": container with ID starting with f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6 not found: ID does not exist" containerID="f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6"
Jul 13 19:25:32.415281 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.415192    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6"} err="failed to get container status \"f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6\": rpc error: code = NotFound desc = could not find container \"f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6\": container with ID starting with f2a44ffa9255309594d6abfdf2f56afcd1d614f587451fbee04a142c6ba66ce6 not found: ID does not exist"
Jul 13 19:25:32.415281 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.415209    2581 scope.go:117] "RemoveContainer" containerID="6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a"
Jul 13 19:25:32.415441 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:25:32.415423    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a\": container with ID starting with 6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a not found: ID does not exist" containerID="6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a"
Jul 13 19:25:32.415482 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.415448    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a"} err="failed to get container status \"6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a\": rpc error: code = NotFound desc = could not find container \"6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a\": container with ID starting with 6603ae443e5bba27e882249ccdae8c9e54e7706d7f4e5d0c4ab8eccfe1cfd41a not found: ID does not exist"
Jul 13 19:25:32.419549 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.419526    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"]
Jul 13 19:25:32.425955 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:32.425923    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf"]
Jul 13 19:25:33.603457 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:25:33.603423    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" path="/var/lib/kubelet/pods/684d5b95-817e-4ec8-ac00-7dbd198f07c3/volumes"
Jul 13 19:26:02.442365 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.442325    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:26:02.638863 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.638826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"]
Jul 13 19:26:02.639301 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639283    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerName="main"
Jul 13 19:26:02.639352 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639306    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerName="main"
Jul 13 19:26:02.639352 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy"
Jul 13 19:26:02.639352 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639336    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy"
Jul 13 19:26:02.639455 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639360    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerName="storage-initializer"
Jul 13 19:26:02.639455 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639370    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerName="storage-initializer"
Jul 13 19:26:02.639455 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639382    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy"
Jul 13 19:26:02.639455 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639392    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy"
Jul 13 19:26:02.639579 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639500    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy"
Jul 13 19:26:02.639579 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639510    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="684d5b95-817e-4ec8-ac00-7dbd198f07c3" containerName="main"
Jul 13 19:26:02.639579 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.639519    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="08bc7aeb-3257-4e2a-b0b9-48f3ecf40f41" containerName="istio-proxy"
Jul 13 19:26:02.642951 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.642929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.647585 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.647560    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-fdfgg\""
Jul 13 19:26:02.648011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.647991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 13 19:26:02.648090 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.647998    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:26:02.648160 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.648145    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:26:02.658287 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.658263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"]
Jul 13 19:26:02.676924 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.676881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:26:02.705142 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.705118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.705266 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.705147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d884944a-3a8c-408c-a859-9743db032481-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.705266 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.705175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.705341 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.705282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxnz\" (UniqueName: \"kubernetes.io/projected/d884944a-3a8c-408c-a859-9743db032481-kube-api-access-twxnz\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.806298 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.806262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.806491 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.806303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d884944a-3a8c-408c-a859-9743db032481-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.806491 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.806347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.806614 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.806511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxnz\" (UniqueName: \"kubernetes.io/projected/d884944a-3a8c-408c-a859-9743db032481-kube-api-access-twxnz\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.806714 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.806694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.806753 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.806705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.808911 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.808886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d884944a-3a8c-408c-a859-9743db032481-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.815698 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.815665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxnz\" (UniqueName: \"kubernetes.io/projected/d884944a-3a8c-408c-a859-9743db032481-kube-api-access-twxnz\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:02.951910 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:02.951872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:03.105175 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:03.105145    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"]
Jul 13 19:26:03.106933 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:26:03.106906    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd884944a_3a8c_408c_a859_9743db032481.slice/crio-3006abbd9559d91ca6c2dd685767990e61ae1fe9d64eaa8465fa141c5910b562 WatchSource:0}: Error finding container 3006abbd9559d91ca6c2dd685767990e61ae1fe9d64eaa8465fa141c5910b562: Status 404 returned error can't find the container with id 3006abbd9559d91ca6c2dd685767990e61ae1fe9d64eaa8465fa141c5910b562
Jul 13 19:26:03.557575 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:03.557536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" event={"ID":"d884944a-3a8c-408c-a859-9743db032481","Type":"ContainerStarted","Data":"2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead"}
Jul 13 19:26:03.557998 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:03.557602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" event={"ID":"d884944a-3a8c-408c-a859-9743db032481","Type":"ContainerStarted","Data":"3006abbd9559d91ca6c2dd685767990e61ae1fe9d64eaa8465fa141c5910b562"}
Jul 13 19:26:04.565244 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:04.565210    2581 generic.go:356] "Generic (PLEG): container finished" podID="d884944a-3a8c-408c-a859-9743db032481" containerID="2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead" exitCode=0
Jul 13 19:26:04.565631 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:04.565255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" event={"ID":"d884944a-3a8c-408c-a859-9743db032481","Type":"ContainerDied","Data":"2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead"}
Jul 13 19:26:05.572024 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:05.571990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" event={"ID":"d884944a-3a8c-408c-a859-9743db032481","Type":"ContainerStarted","Data":"ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084"}
Jul 13 19:26:05.572403 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:05.572117    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:05.591591 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:05.591539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" podStartSLOduration=2.858588526 podStartE2EDuration="3.591521934s" totalImagesPullingTime="0s" totalInitContainerRuntime="732.933408ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:26:05.588830121 +0000 UTC m=+2202.489055513" watchObservedRunningTime="2026-07-13 19:26:05.591521934 +0000 UTC m=+2202.491747325"
Jul 13 19:26:15.912134 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:15.912099    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:26:36.584842 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:36.584812    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:26:37.145199 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:37.145162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:26:37.208234 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:26:37.208200    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:28:15.301305 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:15.301210    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:28:49.451657 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.451617    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:28:49.456980 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.456950    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"]
Jul 13 19:28:49.466602 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:28:49.466560    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:28:49.466602 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:28:49.466602 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:28:49.466602 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084" cmd=["/bin/sleep","15"]
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.466624    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" message="" err=<
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:28:49.466641    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" podUID="d884944a-3a8c-408c-a859-9743db032481" containerName="main" containerID="cri-o://ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084"
Jul 13 19:28:49.466929 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.466688    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" podUID="d884944a-3a8c-408c-a859-9743db032481" containerName="main" containerID="cri-o://ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084" gracePeriod=60
Jul 13 19:28:49.528322 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.528293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:28:49.715009 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.714985    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:28:49.844640 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.844606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-kserve-provision-location\") pod \"d884944a-3a8c-408c-a859-9743db032481\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") "
Jul 13 19:28:49.844640 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.844643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twxnz\" (UniqueName: \"kubernetes.io/projected/d884944a-3a8c-408c-a859-9743db032481-kube-api-access-twxnz\") pod \"d884944a-3a8c-408c-a859-9743db032481\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") "
Jul 13 19:28:49.844949 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.844716    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d884944a-3a8c-408c-a859-9743db032481-tls-certs\") pod \"d884944a-3a8c-408c-a859-9743db032481\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") "
Jul 13 19:28:49.844949 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.844739    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-tokenizer-uds\") pod \"d884944a-3a8c-408c-a859-9743db032481\" (UID: \"d884944a-3a8c-408c-a859-9743db032481\") "
Jul 13 19:28:49.845092 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.845069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d884944a-3a8c-408c-a859-9743db032481" (UID: "d884944a-3a8c-408c-a859-9743db032481"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:28:49.845412 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.845387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d884944a-3a8c-408c-a859-9743db032481" (UID: "d884944a-3a8c-408c-a859-9743db032481"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:28:49.846921 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.846902    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d884944a-3a8c-408c-a859-9743db032481-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d884944a-3a8c-408c-a859-9743db032481" (UID: "d884944a-3a8c-408c-a859-9743db032481"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:28:49.847002 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.846988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d884944a-3a8c-408c-a859-9743db032481-kube-api-access-twxnz" (OuterVolumeSpecName: "kube-api-access-twxnz") pod "d884944a-3a8c-408c-a859-9743db032481" (UID: "d884944a-3a8c-408c-a859-9743db032481"). InnerVolumeSpecName "kube-api-access-twxnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:28:49.945327 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.945292    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d884944a-3a8c-408c-a859-9743db032481-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:28:49.945327 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.945320    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:28:49.945327 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.945329    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d884944a-3a8c-408c-a859-9743db032481-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:28:49.945555 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:49.945341    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxnz\" (UniqueName: \"kubernetes.io/projected/d884944a-3a8c-408c-a859-9743db032481-kube-api-access-twxnz\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:28:50.381393 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.381359    2581 generic.go:356] "Generic (PLEG): container finished" podID="d884944a-3a8c-408c-a859-9743db032481" containerID="ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084" exitCode=0
Jul 13 19:28:50.381571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.381411    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"
Jul 13 19:28:50.381571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.381443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" event={"ID":"d884944a-3a8c-408c-a859-9743db032481","Type":"ContainerDied","Data":"ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084"}
Jul 13 19:28:50.381571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.381499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp" event={"ID":"d884944a-3a8c-408c-a859-9743db032481","Type":"ContainerDied","Data":"3006abbd9559d91ca6c2dd685767990e61ae1fe9d64eaa8465fa141c5910b562"}
Jul 13 19:28:50.381571 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.381519    2581 scope.go:117] "RemoveContainer" containerID="ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084"
Jul 13 19:28:50.393128 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.393106    2581 scope.go:117] "RemoveContainer" containerID="2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead"
Jul 13 19:28:50.401249 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.401230    2581 scope.go:117] "RemoveContainer" containerID="ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084"
Jul 13 19:28:50.401509 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:28:50.401491    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084\": container with ID starting with ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084 not found: ID does not exist" containerID="ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084"
Jul 13 19:28:50.401547 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.401518    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084"} err="failed to get container status \"ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084\": rpc error: code = NotFound desc = could not find container \"ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084\": container with ID starting with ac0b602cfafc6bf29a996c1b6e24385c078f16c00f27bf88cca7272d4a99e084 not found: ID does not exist"
Jul 13 19:28:50.401547 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.401537    2581 scope.go:117] "RemoveContainer" containerID="2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead"
Jul 13 19:28:50.401802 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:28:50.401771    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead\": container with ID starting with 2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead not found: ID does not exist" containerID="2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead"
Jul 13 19:28:50.401846 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.401818    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead"} err="failed to get container status \"2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead\": rpc error: code = NotFound desc = could not find container \"2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead\": container with ID starting with 2afe4356e9e0115f9dd5f9ca80cdf0f7b5354de87b9f68e40b0eb6e03a1f9ead not found: ID does not exist"
Jul 13 19:28:50.404920 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.404897    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"]
Jul 13 19:28:50.409408 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:50.409382    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp"]
Jul 13 19:28:51.331146 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:51.331113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:28:51.384854 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:51.384815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:28:51.608702 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:28:51.605352    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d884944a-3a8c-408c-a859-9743db032481" path="/var/lib/kubelet/pods/d884944a-3a8c-408c-a859-9743db032481/volumes"
Jul 13 19:29:23.719902 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:23.719869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:29:23.727220 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:23.727196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:29:31.041713 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.041673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:29:31.421444 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.421369    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"]
Jul 13 19:29:31.421721 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.421707    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d884944a-3a8c-408c-a859-9743db032481" containerName="storage-initializer"
Jul 13 19:29:31.421762 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.421724    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d884944a-3a8c-408c-a859-9743db032481" containerName="storage-initializer"
Jul 13 19:29:31.421762 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.421738    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d884944a-3a8c-408c-a859-9743db032481" containerName="main"
Jul 13 19:29:31.421762 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.421744    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d884944a-3a8c-408c-a859-9743db032481" containerName="main"
Jul 13 19:29:31.421913 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.421814    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d884944a-3a8c-408c-a859-9743db032481" containerName="main"
Jul 13 19:29:31.424913 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.424891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.427570 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.427520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-jzjzj\""
Jul 13 19:29:31.427934 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.427916    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:29:31.428035 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.427958    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:29:31.428035 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.428004    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 13 19:29:31.437606 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.437585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"]
Jul 13 19:29:31.451869 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.451848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:29:31.460936 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.460915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmft\" (UniqueName: \"kubernetes.io/projected/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kube-api-access-jhmft\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.461026 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.460954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.461075 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.461040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.461075 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.461068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.562730 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.562690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.562954 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.562837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.562954 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.562881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.562954 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.562934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmft\" (UniqueName: \"kubernetes.io/projected/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kube-api-access-jhmft\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.563744 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.563715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.564039 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.564016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.569618 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.569591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.572078 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.572056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmft\" (UniqueName: \"kubernetes.io/projected/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kube-api-access-jhmft\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.733748 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.733719    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:31.886946 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:31.886918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"]
Jul 13 19:29:31.888902 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:29:31.888873    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ecd58d_af53_4206_bc8b_4e1a311a78c2.slice/crio-903b1c8a93be591d6305e5c905fe7acd5513125c1fddd9232ec25c4c7bd4043e WatchSource:0}: Error finding container 903b1c8a93be591d6305e5c905fe7acd5513125c1fddd9232ec25c4c7bd4043e: Status 404 returned error can't find the container with id 903b1c8a93be591d6305e5c905fe7acd5513125c1fddd9232ec25c4c7bd4043e
Jul 13 19:29:32.581292 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:32.581259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" event={"ID":"59ecd58d-af53-4206-bc8b-4e1a311a78c2","Type":"ContainerStarted","Data":"60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97"}
Jul 13 19:29:32.581778 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:32.581308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" event={"ID":"59ecd58d-af53-4206-bc8b-4e1a311a78c2","Type":"ContainerStarted","Data":"903b1c8a93be591d6305e5c905fe7acd5513125c1fddd9232ec25c4c7bd4043e"}
Jul 13 19:29:33.587953 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:33.587915    2581 generic.go:356] "Generic (PLEG): container finished" podID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerID="60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97" exitCode=0
Jul 13 19:29:33.588352 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:33.588008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" event={"ID":"59ecd58d-af53-4206-bc8b-4e1a311a78c2","Type":"ContainerDied","Data":"60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97"}
Jul 13 19:29:34.595182 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:34.595145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" event={"ID":"59ecd58d-af53-4206-bc8b-4e1a311a78c2","Type":"ContainerStarted","Data":"8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4"}
Jul 13 19:29:34.595659 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:34.595227    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:29:34.615317 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:34.615268    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" podStartSLOduration=2.905766861 podStartE2EDuration="3.615250892s" totalImagesPullingTime="0s" totalInitContainerRuntime="709.484031ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:34.613542154 +0000 UTC m=+2411.513767546" watchObservedRunningTime="2026-07-13 19:29:34.615250892 +0000 UTC m=+2411.515476284"
Jul 13 19:29:54.700537 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:29:54.700502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:30:05.603605 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:30:05.603575    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:30:13.073579 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:30:13.073490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:31:26.997918 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:31:26.997825    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:31:36.385648 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:31:36.385610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:24.221843 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:24.221778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:24.466055 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:24.466010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:59.531005 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.530974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:59.546849 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.546812    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"]
Jul 13 19:32:59.556367 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:32:59.556332    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:32:59.556367 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:32:59.556367 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:32:59.556367 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4" cmd=["/bin/sleep","15"]
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.556382    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" message="" err=<
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:32:59.556393    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerName="main" containerID="cri-o://8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4"
Jul 13 19:32:59.556637 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.556437    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerName="main" containerID="cri-o://8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4" gracePeriod=60
Jul 13 19:32:59.599264 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.599237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:32:59.807883 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.807860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:32:59.886386 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.886339    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhmft\" (UniqueName: \"kubernetes.io/projected/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kube-api-access-jhmft\") pod \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") "
Jul 13 19:32:59.886566 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.886422    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tokenizer-uds\") pod \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") "
Jul 13 19:32:59.886566 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.886477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tls-certs\") pod \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") "
Jul 13 19:32:59.886566 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.886502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kserve-provision-location\") pod \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\" (UID: \"59ecd58d-af53-4206-bc8b-4e1a311a78c2\") "
Jul 13 19:32:59.886781 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.886739    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "59ecd58d-af53-4206-bc8b-4e1a311a78c2" (UID: "59ecd58d-af53-4206-bc8b-4e1a311a78c2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:32:59.887252 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.887225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59ecd58d-af53-4206-bc8b-4e1a311a78c2" (UID: "59ecd58d-af53-4206-bc8b-4e1a311a78c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:32:59.888670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.888647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59ecd58d-af53-4206-bc8b-4e1a311a78c2" (UID: "59ecd58d-af53-4206-bc8b-4e1a311a78c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:32:59.888748 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.888733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kube-api-access-jhmft" (OuterVolumeSpecName: "kube-api-access-jhmft") pod "59ecd58d-af53-4206-bc8b-4e1a311a78c2" (UID: "59ecd58d-af53-4206-bc8b-4e1a311a78c2"). InnerVolumeSpecName "kube-api-access-jhmft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:32:59.987085 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.987050    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:32:59.987085 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.987075    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:32:59.987085 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.987084    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmft\" (UniqueName: \"kubernetes.io/projected/59ecd58d-af53-4206-bc8b-4e1a311a78c2-kube-api-access-jhmft\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:32:59.987085 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:32:59.987094    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59ecd58d-af53-4206-bc8b-4e1a311a78c2-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:00.585628 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.585598    2581 generic.go:356] "Generic (PLEG): container finished" podID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerID="8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4" exitCode=0
Jul 13 19:33:00.586145 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.585654    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"
Jul 13 19:33:00.586145 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.585689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" event={"ID":"59ecd58d-af53-4206-bc8b-4e1a311a78c2","Type":"ContainerDied","Data":"8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4"}
Jul 13 19:33:00.586145 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.585740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z" event={"ID":"59ecd58d-af53-4206-bc8b-4e1a311a78c2","Type":"ContainerDied","Data":"903b1c8a93be591d6305e5c905fe7acd5513125c1fddd9232ec25c4c7bd4043e"}
Jul 13 19:33:00.586145 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.585760    2581 scope.go:117] "RemoveContainer" containerID="8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4"
Jul 13 19:33:00.595743 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.595723    2581 scope.go:117] "RemoveContainer" containerID="60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97"
Jul 13 19:33:00.604079 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.604065    2581 scope.go:117] "RemoveContainer" containerID="8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4"
Jul 13 19:33:00.604331 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:33:00.604311    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4\": container with ID starting with 8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4 not found: ID does not exist" containerID="8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4"
Jul 13 19:33:00.604387 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.604339    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4"} err="failed to get container status \"8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4\": rpc error: code = NotFound desc = could not find container \"8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4\": container with ID starting with 8a67b392994472639e6cba879a482385fb8bf80210ae471d143e4114233ac2b4 not found: ID does not exist"
Jul 13 19:33:00.604387 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.604356    2581 scope.go:117] "RemoveContainer" containerID="60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97"
Jul 13 19:33:00.604592 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:33:00.604575    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97\": container with ID starting with 60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97 not found: ID does not exist" containerID="60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97"
Jul 13 19:33:00.604636 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.604599    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97"} err="failed to get container status \"60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97\": rpc error: code = NotFound desc = could not find container \"60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97\": container with ID starting with 60f9b13cf04a585c013564d3c3803167487063e6480680873ed187a1b9242a97 not found: ID does not exist"
Jul 13 19:33:00.608367 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.608344    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"]
Jul 13 19:33:00.614056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:00.614037    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z"]
Jul 13 19:33:01.602901 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:01.602868    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" path="/var/lib/kubelet/pods/59ecd58d-af53-4206-bc8b-4e1a311a78c2/volumes"
Jul 13 19:33:18.000367 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.000329    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"]
Jul 13 19:33:18.000852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.000836    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerName="storage-initializer"
Jul 13 19:33:18.000944 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.000856    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerName="storage-initializer"
Jul 13 19:33:18.000944 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.000884    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerName="main"
Jul 13 19:33:18.000944 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.000893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerName="main"
Jul 13 19:33:18.001101 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.000983    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ecd58d-af53-4206-bc8b-4e1a311a78c2" containerName="main"
Jul 13 19:33:18.004252 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.004229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.008292 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.008268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-xjm54\""
Jul 13 19:33:18.023745 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.023716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"]
Jul 13 19:33:18.042880 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.042847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043004 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.042890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043004 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.042913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043004 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.042940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043004 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.042963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043004 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.042980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043216 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.043049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043216 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.043093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.043216 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.043115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxd7\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-kube-api-access-6cxd7\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.050419 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.050379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:18.143527 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143527 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxd7\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-kube-api-access-6cxd7\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.143767 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.143701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.144164 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.144004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.144164 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.144088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.144164 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.144114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.144327 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.144235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.144370 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.144346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.146142 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.146119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.146373 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.146344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.152346 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.152322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxd7\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-kube-api-access-6cxd7\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.152532 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.152511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-chk28\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.316748 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.316664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:18.457349 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.457326    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"]
Jul 13 19:33:18.459514 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:33:18.459480    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab5480b_3045_4a85_a4c5_1a1222211aeb.slice/crio-d465ba2ea9d4bf5ac67e7aa3c0a2b94f7629ab61860e2b7f071125ccb012756d WatchSource:0}: Error finding container d465ba2ea9d4bf5ac67e7aa3c0a2b94f7629ab61860e2b7f071125ccb012756d: Status 404 returned error can't find the container with id d465ba2ea9d4bf5ac67e7aa3c0a2b94f7629ab61860e2b7f071125ccb012756d
Jul 13 19:33:18.461676 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.461643    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:18.461764 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.461719    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:18.461828 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.461760    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:18.675732 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.675640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28" event={"ID":"6ab5480b-3045-4a85-a4c5-1a1222211aeb","Type":"ContainerStarted","Data":"4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d"}
Jul 13 19:33:18.675732 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.675706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28" event={"ID":"6ab5480b-3045-4a85-a4c5-1a1222211aeb","Type":"ContainerStarted","Data":"d465ba2ea9d4bf5ac67e7aa3c0a2b94f7629ab61860e2b7f071125ccb012756d"}
Jul 13 19:33:18.696800 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:18.696734    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28" podStartSLOduration=1.696716849 podStartE2EDuration="1.696716849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:18.695011788 +0000 UTC m=+2635.595237181" watchObservedRunningTime="2026-07-13 19:33:18.696716849 +0000 UTC m=+2635.596942241"
Jul 13 19:33:19.317467 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:19.317431    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:19.322581 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:19.322557    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:19.680524 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:19.680441    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:19.681360 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:19.681339    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:20.007671 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.007642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:20.639547 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.639515    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"]
Jul 13 19:33:20.643056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.643036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.655004 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.654981    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"]
Jul 13 19:33:20.667736 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.667684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.667736 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.667729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.668011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.667751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.668011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.667883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.668011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.667970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lph\" (UniqueName: \"kubernetes.io/projected/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-kube-api-access-w2lph\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.668011 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.668003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.668225 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.668029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.668225 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.668119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.668331 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.668224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.768670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.768670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.768670 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.768969 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.768969 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lph\" (UniqueName: \"kubernetes.io/projected/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-kube-api-access-w2lph\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.768969 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.768969 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.769202 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.768991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.769202 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.769024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.769202 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.769063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.769202 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.769184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.769428 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.769307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.769428 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.769351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.769428 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.769402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.771379 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.771353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.771751 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.771733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.776442 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.776419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.776517 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.776467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lph\" (UniqueName: \"kubernetes.io/projected/4b0ec9d9-4001-4475-b9a7-9632c3a8118f-kube-api-access-w2lph\") pod \"router-gateway-2-openshift-default-684dc9b4b9-c9txx\" (UID: \"4b0ec9d9-4001-4475-b9a7-9632c3a8118f\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:20.954502 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:20.954467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:21.099629 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.099603    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"]
Jul 13 19:33:21.101318 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:33:21.101286    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0ec9d9_4001_4475_b9a7_9632c3a8118f.slice/crio-6ec821ba1f862f5e4c0384c766f74025be4d3cdfef9445770134895e1280dbc9 WatchSource:0}: Error finding container 6ec821ba1f862f5e4c0384c766f74025be4d3cdfef9445770134895e1280dbc9: Status 404 returned error can't find the container with id 6ec821ba1f862f5e4c0384c766f74025be4d3cdfef9445770134895e1280dbc9
Jul 13 19:33:21.103601 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.103571    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:21.103678 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.103637    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:21.103678 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.103666    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:21.692454 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.692418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" event={"ID":"4b0ec9d9-4001-4475-b9a7-9632c3a8118f","Type":"ContainerStarted","Data":"ad9aba8c8be71778f403fb1758ade57482ef715d5325a7fb9317c609855980b5"}
Jul 13 19:33:21.692886 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.692482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" event={"ID":"4b0ec9d9-4001-4475-b9a7-9632c3a8118f","Type":"ContainerStarted","Data":"6ec821ba1f862f5e4c0384c766f74025be4d3cdfef9445770134895e1280dbc9"}
Jul 13 19:33:21.716358 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.716310    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" podStartSLOduration=1.716294086 podStartE2EDuration="1.716294086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:21.714828064 +0000 UTC m=+2638.615053456" watchObservedRunningTime="2026-07-13 19:33:21.716294086 +0000 UTC m=+2638.616519478"
Jul 13 19:33:21.955283 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:21.955254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:22.639574 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:22.639540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:22.761692 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:22.761663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:22.960341 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:22.960310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:23.273537 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.273458    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"]
Jul 13 19:33:23.276769 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.276749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.279122 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.279096    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 13 19:33:23.279265 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.279124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:33:23.279417 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.279390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-x8klk\""
Jul 13 19:33:23.290061 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.290016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"]
Jul 13 19:33:23.296884 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.296849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657d7\" (UniqueName: \"kubernetes.io/projected/dbcef857-215f-4fa4-8519-e5029d249889-kube-api-access-657d7\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.297048 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.296917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcef857-215f-4fa4-8519-e5029d249889-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.297048 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.296953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbcef857-215f-4fa4-8519-e5029d249889-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.310444 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.310413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:23.397692 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.397650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcef857-215f-4fa4-8519-e5029d249889-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.397692 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.397695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbcef857-215f-4fa4-8519-e5029d249889-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.397952 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.397748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657d7\" (UniqueName: \"kubernetes.io/projected/dbcef857-215f-4fa4-8519-e5029d249889-kube-api-access-657d7\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.398156 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.398137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbcef857-215f-4fa4-8519-e5029d249889-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.400677 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.400649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcef857-215f-4fa4-8519-e5029d249889-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.409218 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.409198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-657d7\" (UniqueName: \"kubernetes.io/projected/dbcef857-215f-4fa4-8519-e5029d249889-kube-api-access-657d7\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb\" (UID: \"dbcef857-215f-4fa4-8519-e5029d249889\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.590911 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.590816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:33:23.601138 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.601109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-x8klk\""
Jul 13 19:33:23.609770 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.609744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:23.705723 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.705696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:23.706917 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.706893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:23.761688 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.761405    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"]
Jul 13 19:33:23.765355 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:33:23.765320    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcef857_215f_4fa4_8519_e5029d249889.slice/crio-e325a68d2d1ded09d9de795abad38170e90ebf5610f0deb3a888da853240102d WatchSource:0}: Error finding container e325a68d2d1ded09d9de795abad38170e90ebf5610f0deb3a888da853240102d: Status 404 returned error can't find the container with id e325a68d2d1ded09d9de795abad38170e90ebf5610f0deb3a888da853240102d
Jul 13 19:33:23.769979 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.769955    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"]
Jul 13 19:33:23.773256 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:23.773195    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28" podUID="6ab5480b-3045-4a85-a4c5-1a1222211aeb" containerName="istio-proxy" containerID="cri-o://4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d" gracePeriod=30
Jul 13 19:33:24.720094 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:24.720060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb" event={"ID":"dbcef857-215f-4fa4-8519-e5029d249889","Type":"ContainerStarted","Data":"bcc9612595c698fba65a2883f5120d0c36c06f7f62c6dd9d9e80efc8f6b0e768"}
Jul 13 19:33:24.720335 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:24.720314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb" event={"ID":"dbcef857-215f-4fa4-8519-e5029d249889","Type":"ContainerStarted","Data":"e325a68d2d1ded09d9de795abad38170e90ebf5610f0deb3a888da853240102d"}
Jul 13 19:33:24.720485 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:24.720470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:33:24.737105 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:24.737034    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb" podStartSLOduration=1.73701118 podStartE2EDuration="1.73701118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:24.735981515 +0000 UTC m=+2641.636206908" watchObservedRunningTime="2026-07-13 19:33:24.73701118 +0000 UTC m=+2641.637236572"
Jul 13 19:33:29.064543 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.064475    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:29.151553 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-socket\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151553 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-data\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-credential-socket\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-envoy\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151622    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istiod-ca-cert\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-certs\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-token\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151705    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cxd7\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-kube-api-access-6cxd7\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.151852 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.151772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-podinfo\") pod \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\" (UID: \"6ab5480b-3045-4a85-a4c5-1a1222211aeb\") "
Jul 13 19:33:29.153985 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.153953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:29.154080 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.153991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:29.154445 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.154414    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:29.154766 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.154726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:33:29.154985 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.154884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:29.156195 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.156168    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:33:29.156288 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.156201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 19:33:29.156288 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.156214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:29.156363 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.156350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-kube-api-access-6cxd7" (OuterVolumeSpecName: "kube-api-access-6cxd7") pod "6ab5480b-3045-4a85-a4c5-1a1222211aeb" (UID: "6ab5480b-3045-4a85-a4c5-1a1222211aeb"). InnerVolumeSpecName "kube-api-access-6cxd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:33:29.253115 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253080    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-podinfo\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253115 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253109    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253115 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253120    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-data\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253361 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253131    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-credential-socket\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253361 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253139    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-envoy\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253361 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253147    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istiod-ca-cert\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253361 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253155    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab5480b-3045-4a85-a4c5-1a1222211aeb-workload-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253361 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253163    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-istio-token\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.253361 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.253171    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxd7\" (UniqueName: \"kubernetes.io/projected/6ab5480b-3045-4a85-a4c5-1a1222211aeb-kube-api-access-6cxd7\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:33:29.753343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.753246    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ab5480b-3045-4a85-a4c5-1a1222211aeb" containerID="4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d" exitCode=0
Jul 13 19:33:29.753343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.753293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"
Jul 13 19:33:29.753343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.753322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28" event={"ID":"6ab5480b-3045-4a85-a4c5-1a1222211aeb","Type":"ContainerDied","Data":"4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d"}
Jul 13 19:33:29.753575 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.753366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28" event={"ID":"6ab5480b-3045-4a85-a4c5-1a1222211aeb","Type":"ContainerDied","Data":"d465ba2ea9d4bf5ac67e7aa3c0a2b94f7629ab61860e2b7f071125ccb012756d"}
Jul 13 19:33:29.753575 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.753386    2581 scope.go:117] "RemoveContainer" containerID="4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d"
Jul 13 19:33:29.762490 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.762472    2581 scope.go:117] "RemoveContainer" containerID="4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d"
Jul 13 19:33:29.762813 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:33:29.762768    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d\": container with ID starting with 4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d not found: ID does not exist" containerID="4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d"
Jul 13 19:33:29.762917 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.762811    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d"} err="failed to get container status \"4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d\": rpc error: code = NotFound desc = could not find container \"4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d\": container with ID starting with 4ecd129e9a69dd2576d86a2072d5f0a66bc422851db56f3a6c1c68de99a9220d not found: ID does not exist"
Jul 13 19:33:29.769393 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.769371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"]
Jul 13 19:33:29.774074 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:29.774052    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28"]
Jul 13 19:33:31.603417 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:31.603375    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab5480b-3045-4a85-a4c5-1a1222211aeb" path="/var/lib/kubelet/pods/6ab5480b-3045-4a85-a4c5-1a1222211aeb/volumes"
Jul 13 19:33:32.776100 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:32.776066    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b0ec9d9-4001-4475-b9a7-9632c3a8118f" containerID="ad9aba8c8be71778f403fb1758ade57482ef715d5325a7fb9317c609855980b5" exitCode=0
Jul 13 19:33:32.776511 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:32.776109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" event={"ID":"4b0ec9d9-4001-4475-b9a7-9632c3a8118f","Type":"ContainerDied","Data":"ad9aba8c8be71778f403fb1758ade57482ef715d5325a7fb9317c609855980b5"}
Jul 13 19:33:32.776511 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:32.776434    2581 scope.go:117] "RemoveContainer" containerID="ad9aba8c8be71778f403fb1758ade57482ef715d5325a7fb9317c609855980b5"
Jul 13 19:33:32.777382 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:32.777354    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:32.777454 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:32.777420    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:32.777454 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:32.777448    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:33:33.783276 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:33.783245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" event={"ID":"4b0ec9d9-4001-4475-b9a7-9632c3a8118f","Type":"ContainerStarted","Data":"6bcb3b6e3a232c9cc80084e720116dfa7cf63d79c8515abcb1adc6028af20afd"}
Jul 13 19:33:33.955374 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:33.955342    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:33.956576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:33.956545    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" podUID="4b0ec9d9-4001-4475-b9a7-9632c3a8118f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 19:33:34.955616 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:34.955570    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" podUID="4b0ec9d9-4001-4475-b9a7-9632c3a8118f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 19:33:35.955392 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:35.955353    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" podUID="4b0ec9d9-4001-4475-b9a7-9632c3a8118f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 19:33:36.955102 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:36.955060    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" podUID="4b0ec9d9-4001-4475-b9a7-9632c3a8118f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 19:33:37.955381 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:37.955339    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx" podUID="4b0ec9d9-4001-4475-b9a7-9632c3a8118f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 13 19:33:38.706045 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:38.706002    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:38.959374 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:38.959297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:38.960456 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:38.960435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx"
Jul 13 19:33:50.392551 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.392518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:50.870211 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.870171    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"]
Jul 13 19:33:50.870526 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.870514    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab5480b-3045-4a85-a4c5-1a1222211aeb" containerName="istio-proxy"
Jul 13 19:33:50.870574 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.870528    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab5480b-3045-4a85-a4c5-1a1222211aeb" containerName="istio-proxy"
Jul 13 19:33:50.870609 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.870585    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab5480b-3045-4a85-a4c5-1a1222211aeb" containerName="istio-proxy"
Jul 13 19:33:50.873708 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.873687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:50.876060 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.876038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-b9zm5\""
Jul 13 19:33:50.876163 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.876039    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:33:50.876427 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.876410    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 13 19:33:50.882984 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.882963    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"]
Jul 13 19:33:50.928822 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.928770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:50.928997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.928831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b0b64c-9518-41d7-92a9-e9faddc4b031-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:50.928997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.928888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:50.928997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:50.928910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6gn\" (UniqueName: \"kubernetes.io/projected/46b0b64c-9518-41d7-92a9-e9faddc4b031-kube-api-access-zk6gn\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.029693 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.029660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.029885 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.029710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b0b64c-9518-41d7-92a9-e9faddc4b031-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.029885 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.029751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.029885 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.029775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6gn\" (UniqueName: \"kubernetes.io/projected/46b0b64c-9518-41d7-92a9-e9faddc4b031-kube-api-access-zk6gn\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.030184 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.030159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.030272 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.030169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.032443 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.032422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b0b64c-9518-41d7-92a9-e9faddc4b031-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.037125 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.037106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6gn\" (UniqueName: \"kubernetes.io/projected/46b0b64c-9518-41d7-92a9-e9faddc4b031-kube-api-access-zk6gn\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.183965 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.183864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:51.320353 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.320329    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"]
Jul 13 19:33:51.322689 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:33:51.322661    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b0b64c_9518_41d7_92a9_e9faddc4b031.slice/crio-3359675da2cd9c76a4f07cce49ce3887461e034ee008a1a58b164a68b07dc0b1 WatchSource:0}: Error finding container 3359675da2cd9c76a4f07cce49ce3887461e034ee008a1a58b164a68b07dc0b1: Status 404 returned error can't find the container with id 3359675da2cd9c76a4f07cce49ce3887461e034ee008a1a58b164a68b07dc0b1
Jul 13 19:33:51.883429 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.883381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" event={"ID":"46b0b64c-9518-41d7-92a9-e9faddc4b031","Type":"ContainerStarted","Data":"a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1"}
Jul 13 19:33:51.883859 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:51.883452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" event={"ID":"46b0b64c-9518-41d7-92a9-e9faddc4b031","Type":"ContainerStarted","Data":"3359675da2cd9c76a4f07cce49ce3887461e034ee008a1a58b164a68b07dc0b1"}
Jul 13 19:33:52.626945 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:52.626908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:52.727817 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:52.727753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:33:52.891331 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:52.891232    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerID="a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1" exitCode=0
Jul 13 19:33:52.891703 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:52.891320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" event={"ID":"46b0b64c-9518-41d7-92a9-e9faddc4b031","Type":"ContainerDied","Data":"a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1"}
Jul 13 19:33:53.899038 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:53.899001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" event={"ID":"46b0b64c-9518-41d7-92a9-e9faddc4b031","Type":"ContainerStarted","Data":"36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d"}
Jul 13 19:33:53.899426 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:53.899089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:33:53.920588 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:53.920539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" podStartSLOduration=3.146317093 podStartE2EDuration="3.920523223s" totalImagesPullingTime="0s" totalInitContainerRuntime="774.20613ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:53.917643969 +0000 UTC m=+2670.817869361" watchObservedRunningTime="2026-07-13 19:33:53.920523223 +0000 UTC m=+2670.820748614"
Jul 13 19:33:56.738356 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:33:56.738326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb"
Jul 13 19:34:00.240272 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:34:00.240227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:34:23.749931 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:34:23.749898    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:34:23.759805 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:34:23.759768    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:34:24.906074 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:34:24.906045    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:34:44.117444 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:34:44.117401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:34:44.774522 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:34:44.774493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:36:03.144167 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:36:03.144089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:37:12.083170 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.083111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:37:12.098235 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.098198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"]
Jul 13 19:37:12.109026 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:37:12.108981    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:37:12.109026 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:37:12.109026 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:37:12.109026 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d" cmd=["/bin/sleep","15"]
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.109050    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" message="" err=<
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:37:12.109067    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerName="main" containerID="cri-o://36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d"
Jul 13 19:37:12.109325 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.109126    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerName="main" containerID="cri-o://36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d" gracePeriod=60
Jul 13 19:37:12.149569 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.149518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:37:12.365344 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.365323    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:37:12.465719 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.465692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk6gn\" (UniqueName: \"kubernetes.io/projected/46b0b64c-9518-41d7-92a9-e9faddc4b031-kube-api-access-zk6gn\") pod \"46b0b64c-9518-41d7-92a9-e9faddc4b031\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") "
Jul 13 19:37:12.465938 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.465740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b0b64c-9518-41d7-92a9-e9faddc4b031-tls-certs\") pod \"46b0b64c-9518-41d7-92a9-e9faddc4b031\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") "
Jul 13 19:37:12.465938 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.465780    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-tokenizer-uds\") pod \"46b0b64c-9518-41d7-92a9-e9faddc4b031\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") "
Jul 13 19:37:12.465938 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.465829    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-kserve-provision-location\") pod \"46b0b64c-9518-41d7-92a9-e9faddc4b031\" (UID: \"46b0b64c-9518-41d7-92a9-e9faddc4b031\") "
Jul 13 19:37:12.466162 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.466130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "46b0b64c-9518-41d7-92a9-e9faddc4b031" (UID: "46b0b64c-9518-41d7-92a9-e9faddc4b031"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:12.466593 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.466549    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46b0b64c-9518-41d7-92a9-e9faddc4b031" (UID: "46b0b64c-9518-41d7-92a9-e9faddc4b031"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:12.468073 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.468043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b0b64c-9518-41d7-92a9-e9faddc4b031-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46b0b64c-9518-41d7-92a9-e9faddc4b031" (UID: "46b0b64c-9518-41d7-92a9-e9faddc4b031"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:37:12.468167 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.468097    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b0b64c-9518-41d7-92a9-e9faddc4b031-kube-api-access-zk6gn" (OuterVolumeSpecName: "kube-api-access-zk6gn") pod "46b0b64c-9518-41d7-92a9-e9faddc4b031" (UID: "46b0b64c-9518-41d7-92a9-e9faddc4b031"). InnerVolumeSpecName "kube-api-access-zk6gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:37:12.566717 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.566687    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk6gn\" (UniqueName: \"kubernetes.io/projected/46b0b64c-9518-41d7-92a9-e9faddc4b031-kube-api-access-zk6gn\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:37:12.566717 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.566714    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b0b64c-9518-41d7-92a9-e9faddc4b031-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:37:12.566717 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.566723    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:37:12.566936 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.566731    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b0b64c-9518-41d7-92a9-e9faddc4b031-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:37:12.871023 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.870984    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerID="36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d" exitCode=0
Jul 13 19:37:12.871023 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.871021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" event={"ID":"46b0b64c-9518-41d7-92a9-e9faddc4b031","Type":"ContainerDied","Data":"36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d"}
Jul 13 19:37:12.871299 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.871044    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"
Jul 13 19:37:12.871299 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.871064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz" event={"ID":"46b0b64c-9518-41d7-92a9-e9faddc4b031","Type":"ContainerDied","Data":"3359675da2cd9c76a4f07cce49ce3887461e034ee008a1a58b164a68b07dc0b1"}
Jul 13 19:37:12.871299 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.871084    2581 scope.go:117] "RemoveContainer" containerID="36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d"
Jul 13 19:37:12.880781 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.880765    2581 scope.go:117] "RemoveContainer" containerID="a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1"
Jul 13 19:37:12.888625 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.888607    2581 scope.go:117] "RemoveContainer" containerID="36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d"
Jul 13 19:37:12.888892 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:37:12.888874    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d\": container with ID starting with 36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d not found: ID does not exist" containerID="36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d"
Jul 13 19:37:12.888950 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.888901    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d"} err="failed to get container status \"36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d\": rpc error: code = NotFound desc = could not find container \"36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d\": container with ID starting with 36b41ed6a36fb2c9ca1b98bd9fe4b5e03991ca5169a983aa2d26929099d31b4d not found: ID does not exist"
Jul 13 19:37:12.888950 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.888920    2581 scope.go:117] "RemoveContainer" containerID="a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1"
Jul 13 19:37:12.889199 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:37:12.889179    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1\": container with ID starting with a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1 not found: ID does not exist" containerID="a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1"
Jul 13 19:37:12.889257 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.889208    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1"} err="failed to get container status \"a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1\": rpc error: code = NotFound desc = could not find container \"a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1\": container with ID starting with a1884b41f9c36a75386a713308e65a13f1aa6f919768fc3ec476542b4e1b3db1 not found: ID does not exist"
Jul 13 19:37:12.893278 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.893255    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"]
Jul 13 19:37:12.897925 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:12.897900    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz"]
Jul 13 19:37:13.605472 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:13.605433    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" path="/var/lib/kubelet/pods/46b0b64c-9518-41d7-92a9-e9faddc4b031/volumes"
Jul 13 19:37:53.005049 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:37:53.005012    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:39:23.780870 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:39:23.780843    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:39:23.791633 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:39:23.791611    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:39:26.215152 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:39:26.215120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:06.349518 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:40:06.349474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:06.449533 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:40:06.449491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:40:46.666674 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:40:46.666643    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:41:30.242193 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.242157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:41:30.343654 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.343615    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:41:30.449857 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.449818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:41:30.493742 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.493660    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"]
Jul 13 19:41:30.494193 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.494174    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerName="main"
Jul 13 19:41:30.494271 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.494197    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerName="main"
Jul 13 19:41:30.494271 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.494213    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerName="storage-initializer"
Jul 13 19:41:30.494271 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.494223    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerName="storage-initializer"
Jul 13 19:41:30.494373 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.494304    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b0b64c-9518-41d7-92a9-e9faddc4b031" containerName="main"
Jul 13 19:41:30.503088 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.503062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.505883 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.505860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:41:30.506003 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.505881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 13 19:41:30.506003 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.505860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-9c6ks\""
Jul 13 19:41:30.514343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.514320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"]
Jul 13 19:41:30.539766 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.539738    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 13 19:41:30.542947 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.542916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:41:30.543076 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.543046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.543336 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.543317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.543393 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.543365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5da50ea-069d-42a0-b11d-93cc52ad767c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.543505 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.543485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.543544 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.543528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwvq\" (UniqueName: \"kubernetes.io/projected/f5da50ea-069d-42a0-b11d-93cc52ad767c-kube-api-access-rfwvq\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.545550 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.545529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-wwgss\""
Jul 13 19:41:30.554839 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.554816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 13 19:41:30.644463 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.644463 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.644816 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwvq\" (UniqueName: \"kubernetes.io/projected/f5da50ea-069d-42a0-b11d-93cc52ad767c-kube-api-access-rfwvq\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.644816 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.644816 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.644816 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.644816 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5da50ea-069d-42a0-b11d-93cc52ad767c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.644816 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.645103 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.645103 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.645103 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.645103 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xm8\" (UniqueName: \"kubernetes.io/projected/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kube-api-access-58xm8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.645103 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.644991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.647245 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.647227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5da50ea-069d-42a0-b11d-93cc52ad767c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.653651 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.653623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwvq\" (UniqueName: \"kubernetes.io/projected/f5da50ea-069d-42a0-b11d-93cc52ad767c-kube-api-access-rfwvq\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.746065 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.745996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746065 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746065 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746339 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746339 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746466 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746466 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746466 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746466 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xm8\" (UniqueName: \"kubernetes.io/projected/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kube-api-access-58xm8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746678 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.746733 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.746691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.748415 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.748393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.748695 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.748677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.753580 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.753559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xm8\" (UniqueName: \"kubernetes.io/projected/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kube-api-access-58xm8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.812693 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.812666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:30.853169 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.853139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:41:30.964416 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:30.964390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"]
Jul 13 19:41:30.966704 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:41:30.966674    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5da50ea_069d_42a0_b11d_93cc52ad767c.slice/crio-3140f654be5b883937927174e92c4037df53a5e3a8deb64fa8881aed299ba8a6 WatchSource:0}: Error finding container 3140f654be5b883937927174e92c4037df53a5e3a8deb64fa8881aed299ba8a6: Status 404 returned error can't find the container with id 3140f654be5b883937927174e92c4037df53a5e3a8deb64fa8881aed299ba8a6
Jul 13 19:41:31.008610 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:31.008579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 13 19:41:31.008717 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:41:31.008652    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420cb406_c3d5_42e3_a98d_2ca99c63e0ad.slice/crio-1e6ee69eaba8d5f3399d80a1aab69efcbffafdcb8a93c9fa85ad5b3202c9cc3b WatchSource:0}: Error finding container 1e6ee69eaba8d5f3399d80a1aab69efcbffafdcb8a93c9fa85ad5b3202c9cc3b: Status 404 returned error can't find the container with id 1e6ee69eaba8d5f3399d80a1aab69efcbffafdcb8a93c9fa85ad5b3202c9cc3b
Jul 13 19:41:31.135249 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:31.135211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"420cb406-c3d5-42e3-a98d-2ca99c63e0ad","Type":"ContainerStarted","Data":"70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5"}
Jul 13 19:41:31.135444 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:31.135273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"420cb406-c3d5-42e3-a98d-2ca99c63e0ad","Type":"ContainerStarted","Data":"1e6ee69eaba8d5f3399d80a1aab69efcbffafdcb8a93c9fa85ad5b3202c9cc3b"}
Jul 13 19:41:31.136903 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:31.136873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" event={"ID":"f5da50ea-069d-42a0-b11d-93cc52ad767c","Type":"ContainerStarted","Data":"02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e"}
Jul 13 19:41:31.137083 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:31.136925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" event={"ID":"f5da50ea-069d-42a0-b11d-93cc52ad767c","Type":"ContainerStarted","Data":"3140f654be5b883937927174e92c4037df53a5e3a8deb64fa8881aed299ba8a6"}
Jul 13 19:41:32.146208 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:32.146108    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerID="02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e" exitCode=0
Jul 13 19:41:32.146754 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:32.146187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" event={"ID":"f5da50ea-069d-42a0-b11d-93cc52ad767c","Type":"ContainerDied","Data":"02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e"}
Jul 13 19:41:33.159037 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:33.158991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" event={"ID":"f5da50ea-069d-42a0-b11d-93cc52ad767c","Type":"ContainerStarted","Data":"04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb"}
Jul 13 19:41:33.159545 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:33.159252    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:41:33.182036 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:33.181987    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" podStartSLOduration=2.432114913 podStartE2EDuration="3.181972249s" totalImagesPullingTime="0s" totalInitContainerRuntime="749.857336ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:41:33.179075976 +0000 UTC m=+3130.079301359" watchObservedRunningTime="2026-07-13 19:41:33.181972249 +0000 UTC m=+3130.082197641"
Jul 13 19:41:35.171887 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:35.171852    2581 generic.go:356] "Generic (PLEG): container finished" podID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerID="70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5" exitCode=0
Jul 13 19:41:35.172274 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:35.171918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"420cb406-c3d5-42e3-a98d-2ca99c63e0ad","Type":"ContainerDied","Data":"70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5"}
Jul 13 19:41:35.173195 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:35.173179    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 19:41:55.049124 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:41:55.049029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:42:03.355630 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:42:03.355585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"420cb406-c3d5-42e3-a98d-2ca99c63e0ad","Type":"ContainerStarted","Data":"b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7"}
Jul 13 19:42:03.378697 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:42:03.378645    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.492542807 podStartE2EDuration="33.378629383s" totalImagesPullingTime="27.760604048s" totalInitContainerRuntime="3.125482528s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:42:03.375321428 +0000 UTC m=+3160.275546820" watchObservedRunningTime="2026-07-13 19:42:03.378629383 +0000 UTC m=+3160.278854775"
Jul 13 19:42:04.168385 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:42:04.168351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:44:23.813019 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:44:23.812994    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:44:23.825221 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:44:23.825199    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:45:10.078139 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:45:10.078106    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:47:55.119614 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:47:55.119545    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 19:48:03.253919 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:03.253883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:48:10.246626 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:10.246593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:48:21.376353 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.376318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:48:21.404901 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.404870    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 13 19:48:21.408116 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.408101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.410056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.410033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh\""
Jul 13 19:48:21.410599 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.410579    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 13 19:48:21.426971 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.426947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 13 19:48:21.443605 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.443560    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:48:21.466132 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.466107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.466264 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.466139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.466264 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.466159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zc7x\" (UniqueName: \"kubernetes.io/projected/2a700b56-6140-4f86-817f-877ba9bd95ce-kube-api-access-7zc7x\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.466264 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.466213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.466264 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.466233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a700b56-6140-4f86-817f-877ba9bd95ce-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.466448 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.466306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.566740 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.566706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.566740 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.566746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.567015 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.566763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zc7x\" (UniqueName: \"kubernetes.io/projected/2a700b56-6140-4f86-817f-877ba9bd95ce-kube-api-access-7zc7x\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.567015 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.566825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.567015 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.566850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a700b56-6140-4f86-817f-877ba9bd95ce-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.567015 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.566889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.567246 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.567177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.567246 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.567206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.567343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.567244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.569090 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.569069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a700b56-6140-4f86-817f-877ba9bd95ce-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.569463 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.569439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a700b56-6140-4f86-817f-877ba9bd95ce-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.574962 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.574937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zc7x\" (UniqueName: \"kubernetes.io/projected/2a700b56-6140-4f86-817f-877ba9bd95ce-kube-api-access-7zc7x\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"2a700b56-6140-4f86-817f-877ba9bd95ce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:48:21.653766 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:21.653685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:48:57.105042 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:48:57.105007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:49:21.529343 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.529308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:49:21.537282 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.537247    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"]
Jul 13 19:49:21.546655 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:49:21.546623    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:49:21.546655 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:49:21.546655 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:49:21.546655 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb" cmd=["/bin/sleep","15"]
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.546669    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" message="" err=<
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:49:21.546682    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerName="main" containerID="cri-o://04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb"
Jul 13 19:49:21.546968 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.546716    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerName="main" containerID="cri-o://04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb" gracePeriod=60
Jul 13 19:49:21.590419 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.590394    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:49:21.818340 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.818313    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:49:21.986026 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.985989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-tokenizer-uds\") pod \"f5da50ea-069d-42a0-b11d-93cc52ad767c\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") "
Jul 13 19:49:21.986221 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.986113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5da50ea-069d-42a0-b11d-93cc52ad767c-tls-certs\") pod \"f5da50ea-069d-42a0-b11d-93cc52ad767c\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") "
Jul 13 19:49:21.986221 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.986152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-kserve-provision-location\") pod \"f5da50ea-069d-42a0-b11d-93cc52ad767c\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") "
Jul 13 19:49:21.986221 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.986186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfwvq\" (UniqueName: \"kubernetes.io/projected/f5da50ea-069d-42a0-b11d-93cc52ad767c-kube-api-access-rfwvq\") pod \"f5da50ea-069d-42a0-b11d-93cc52ad767c\" (UID: \"f5da50ea-069d-42a0-b11d-93cc52ad767c\") "
Jul 13 19:49:21.986380 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.986319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f5da50ea-069d-42a0-b11d-93cc52ad767c" (UID: "f5da50ea-069d-42a0-b11d-93cc52ad767c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:21.986447 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.986414    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:21.986868 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.986719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5da50ea-069d-42a0-b11d-93cc52ad767c" (UID: "f5da50ea-069d-42a0-b11d-93cc52ad767c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:21.988347 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.988320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5da50ea-069d-42a0-b11d-93cc52ad767c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5da50ea-069d-42a0-b11d-93cc52ad767c" (UID: "f5da50ea-069d-42a0-b11d-93cc52ad767c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:49:21.988473 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:21.988456    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5da50ea-069d-42a0-b11d-93cc52ad767c-kube-api-access-rfwvq" (OuterVolumeSpecName: "kube-api-access-rfwvq") pod "f5da50ea-069d-42a0-b11d-93cc52ad767c" (UID: "f5da50ea-069d-42a0-b11d-93cc52ad767c"). InnerVolumeSpecName "kube-api-access-rfwvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:49:22.087263 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.087179    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5da50ea-069d-42a0-b11d-93cc52ad767c-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:22.087263 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.087208    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5da50ea-069d-42a0-b11d-93cc52ad767c-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:22.087263 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.087218    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwvq\" (UniqueName: \"kubernetes.io/projected/f5da50ea-069d-42a0-b11d-93cc52ad767c-kube-api-access-rfwvq\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:22.607348 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.607317    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerID="04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb" exitCode=0
Jul 13 19:49:22.607865 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.607378    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"
Jul 13 19:49:22.607865 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.607402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" event={"ID":"f5da50ea-069d-42a0-b11d-93cc52ad767c","Type":"ContainerDied","Data":"04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb"}
Jul 13 19:49:22.607865 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.607468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn" event={"ID":"f5da50ea-069d-42a0-b11d-93cc52ad767c","Type":"ContainerDied","Data":"3140f654be5b883937927174e92c4037df53a5e3a8deb64fa8881aed299ba8a6"}
Jul 13 19:49:22.607865 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.607489    2581 scope.go:117] "RemoveContainer" containerID="04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb"
Jul 13 19:49:22.617943 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.617922    2581 scope.go:117] "RemoveContainer" containerID="02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e"
Jul 13 19:49:22.630059 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.629725    2581 scope.go:117] "RemoveContainer" containerID="04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb"
Jul 13 19:49:22.632218 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.632186    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"]
Jul 13 19:49:22.637682 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:49:22.632525    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb\": container with ID starting with 04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb not found: ID does not exist" containerID="04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb"
Jul 13 19:49:22.637682 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.632575    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb"} err="failed to get container status \"04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb\": rpc error: code = NotFound desc = could not find container \"04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb\": container with ID starting with 04419024ef0e90ee9de7b92d45dc2ba9c82fd5a5a0af3a56b7efa4d248039fcb not found: ID does not exist"
Jul 13 19:49:22.637682 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.632606    2581 scope.go:117] "RemoveContainer" containerID="02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e"
Jul 13 19:49:22.637682 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.636880    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn"]
Jul 13 19:49:22.638249 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:49:22.638223    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e\": container with ID starting with 02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e not found: ID does not exist" containerID="02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e"
Jul 13 19:49:22.638323 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:22.638260    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e"} err="failed to get container status \"02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e\": rpc error: code = NotFound desc = could not find container \"02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e\": container with ID starting with 02307ad7d14b584a0b7d012978192ff35ffffdd87c6cf6a670a49c7c2279692e not found: ID does not exist"
Jul 13 19:49:23.603695 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:23.603662    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" path="/var/lib/kubelet/pods/f5da50ea-069d-42a0-b11d-93cc52ad767c/volumes"
Jul 13 19:49:23.845344 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:23.845317    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:49:23.858539 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:23.858479    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:49:37.718575 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:37.718539    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 13 19:49:52.741854 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:52.741815    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerName="main" containerID="cri-o://b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7" gracePeriod=45
Jul 13 19:49:53.392756 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.392730    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:49:53.461873 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.461840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-home\") pod \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") "
Jul 13 19:49:53.462046 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.461886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-dshm\") pod \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") "
Jul 13 19:49:53.462046 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.461935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tmp-dir\") pod \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") "
Jul 13 19:49:53.462046 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.461960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58xm8\" (UniqueName: \"kubernetes.io/projected/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kube-api-access-58xm8\") pod \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") "
Jul 13 19:49:53.462046 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.462007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-model-cache\") pod \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") "
Jul 13 19:49:53.462272 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.462049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kserve-provision-location\") pod \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") "
Jul 13 19:49:53.462272 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.462120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tls-certs\") pod \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\" (UID: \"420cb406-c3d5-42e3-a98d-2ca99c63e0ad\") "
Jul 13 19:49:53.462378 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.462304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "420cb406-c3d5-42e3-a98d-2ca99c63e0ad" (UID: "420cb406-c3d5-42e3-a98d-2ca99c63e0ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:53.462463 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.462429    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:53.462522 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.462502    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-home" (OuterVolumeSpecName: "home") pod "420cb406-c3d5-42e3-a98d-2ca99c63e0ad" (UID: "420cb406-c3d5-42e3-a98d-2ca99c63e0ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:53.464229 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.464198    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kube-api-access-58xm8" (OuterVolumeSpecName: "kube-api-access-58xm8") pod "420cb406-c3d5-42e3-a98d-2ca99c63e0ad" (UID: "420cb406-c3d5-42e3-a98d-2ca99c63e0ad"). InnerVolumeSpecName "kube-api-access-58xm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:49:53.464363 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.464349    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "420cb406-c3d5-42e3-a98d-2ca99c63e0ad" (UID: "420cb406-c3d5-42e3-a98d-2ca99c63e0ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:49:53.464556 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.464530    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-dshm" (OuterVolumeSpecName: "dshm") pod "420cb406-c3d5-42e3-a98d-2ca99c63e0ad" (UID: "420cb406-c3d5-42e3-a98d-2ca99c63e0ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:53.479942 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.479912    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "420cb406-c3d5-42e3-a98d-2ca99c63e0ad" (UID: "420cb406-c3d5-42e3-a98d-2ca99c63e0ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:53.521390 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.521362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "420cb406-c3d5-42e3-a98d-2ca99c63e0ad" (UID: "420cb406-c3d5-42e3-a98d-2ca99c63e0ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:53.562854 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.562828    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:53.562854 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.562850    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:53.562854 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.562858    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:53.563029 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.562865    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:53.563029 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.562873    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58xm8\" (UniqueName: \"kubernetes.io/projected/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kube-api-access-58xm8\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:53.563029 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.562881    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/420cb406-c3d5-42e3-a98d-2ca99c63e0ad-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:49:53.762886 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.762772    2581 generic.go:356] "Generic (PLEG): container finished" podID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerID="b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7" exitCode=0
Jul 13 19:49:53.762886 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.762823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"420cb406-c3d5-42e3-a98d-2ca99c63e0ad","Type":"ContainerDied","Data":"b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7"}
Jul 13 19:49:53.762886 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.762839    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 13 19:49:53.762886 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.762880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"420cb406-c3d5-42e3-a98d-2ca99c63e0ad","Type":"ContainerDied","Data":"1e6ee69eaba8d5f3399d80a1aab69efcbffafdcb8a93c9fa85ad5b3202c9cc3b"}
Jul 13 19:49:53.763642 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.762906    2581 scope.go:117] "RemoveContainer" containerID="b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7"
Jul 13 19:49:53.771351 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.771327    2581 scope.go:117] "RemoveContainer" containerID="70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5"
Jul 13 19:49:53.780558 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.780522    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 13 19:49:53.781209 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.781156    2581 scope.go:117] "RemoveContainer" containerID="b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7"
Jul 13 19:49:53.781494 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:49:53.781461    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7\": container with ID starting with b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7 not found: ID does not exist" containerID="b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7"
Jul 13 19:49:53.781560 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.781486    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7"} err="failed to get container status \"b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7\": rpc error: code = NotFound desc = could not find container \"b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7\": container with ID starting with b5a801dce4cf1075473428c21b217202618f9701512491910fc48a940f0cd0f7 not found: ID does not exist"
Jul 13 19:49:53.781560 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.781504    2581 scope.go:117] "RemoveContainer" containerID="70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5"
Jul 13 19:49:53.781780 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:49:53.781749    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5\": container with ID starting with 70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5 not found: ID does not exist" containerID="70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5"
Jul 13 19:49:53.781870 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.781857    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5"} err="failed to get container status \"70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5\": rpc error: code = NotFound desc = could not find container \"70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5\": container with ID starting with 70bcdb7d30ab584110088375bf28da470f592dd9afb47ebf9e726f4e71c587f5 not found: ID does not exist"
Jul 13 19:49:53.783068 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:53.783047    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 13 19:49:55.602858 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:49:55.602825    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" path="/var/lib/kubelet/pods/420cb406-c3d5-42e3-a98d-2ca99c63e0ad/volumes"
Jul 13 19:50:22.598331 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.598296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:50:22.812215 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812177    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"]
Jul 13 19:50:22.812623 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812604    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerName="storage-initializer"
Jul 13 19:50:22.812623 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812625    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerName="storage-initializer"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812645    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerName="storage-initializer"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812653    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerName="storage-initializer"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812663    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerName="main"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812672    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerName="main"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812693    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerName="main"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812702    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerName="main"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812778    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="420cb406-c3d5-42e3-a98d-2ca99c63e0ad" containerName="main"
Jul 13 19:50:22.812811 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.812807    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5da50ea-069d-42a0-b11d-93cc52ad767c" containerName="main"
Jul 13 19:50:22.815920 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.815898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:22.818102 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.818073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 13 19:50:22.818102 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.818075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-8t9kl\""
Jul 13 19:50:22.826136 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.826114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"]
Jul 13 19:50:22.914583 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.914467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f71108-4328-480d-a2cb-3ef98d55c073-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:22.914583 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.914535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rrk\" (UniqueName: \"kubernetes.io/projected/21f71108-4328-480d-a2cb-3ef98d55c073-kube-api-access-g8rrk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:22.914583 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:22.914580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21f71108-4328-480d-a2cb-3ef98d55c073-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.015600 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.015547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f71108-4328-480d-a2cb-3ef98d55c073-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.015600 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.015611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rrk\" (UniqueName: \"kubernetes.io/projected/21f71108-4328-480d-a2cb-3ef98d55c073-kube-api-access-g8rrk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.015875 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.015646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21f71108-4328-480d-a2cb-3ef98d55c073-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.016061 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.016034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21f71108-4328-480d-a2cb-3ef98d55c073-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.018256 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.018232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f71108-4328-480d-a2cb-3ef98d55c073-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.024679 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.024650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rrk\" (UniqueName: \"kubernetes.io/projected/21f71108-4328-480d-a2cb-3ef98d55c073-kube-api-access-g8rrk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.125536 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.125492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.267474 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.267443    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"]
Jul 13 19:50:23.270021 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:50:23.269989    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f71108_4328_480d_a2cb_3ef98d55c073.slice/crio-db3520d7bdd0cb6901861c1c343ad4c773f853c26f07befd4e2ed170e95202ea WatchSource:0}: Error finding container db3520d7bdd0cb6901861c1c343ad4c773f853c26f07befd4e2ed170e95202ea: Status 404 returned error can't find the container with id db3520d7bdd0cb6901861c1c343ad4c773f853c26f07befd4e2ed170e95202ea
Jul 13 19:50:23.908192 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.908155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" event={"ID":"21f71108-4328-480d-a2cb-3ef98d55c073","Type":"ContainerStarted","Data":"5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb"}
Jul 13 19:50:23.908740 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.908203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" event={"ID":"21f71108-4328-480d-a2cb-3ef98d55c073","Type":"ContainerStarted","Data":"db3520d7bdd0cb6901861c1c343ad4c773f853c26f07befd4e2ed170e95202ea"}
Jul 13 19:50:23.908740 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.908278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:50:23.927216 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:23.927168    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" podStartSLOduration=1.927156552 podStartE2EDuration="1.927156552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:50:23.925309473 +0000 UTC m=+3660.825534855" watchObservedRunningTime="2026-07-13 19:50:23.927156552 +0000 UTC m=+3660.827381944"
Jul 13 19:50:24.236174 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:24.236140    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:50:24.419558 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:50:24.419516    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 19:50:24.912433 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:24.912407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:50:54.913997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:50:54.913914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:51:22.983028 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:51:22.982995    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:51:58.457846 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:51:58.457815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:52:27.913815 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:52:27.913700    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 19:52:28.528077 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:28.528044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:52:56.244419 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.244378    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"]
Jul 13 19:52:56.254110 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:52:56.254074    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:52:56.254110 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:52:56.254110 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:52:56.254110 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb" cmd=["/bin/sleep","15"]
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.254128    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" message="" err=<
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:52:56.254141    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" podUID="21f71108-4328-480d-a2cb-3ef98d55c073" containerName="main" containerID="cri-o://5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb"
Jul 13 19:52:56.254451 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.254177    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" podUID="21f71108-4328-480d-a2cb-3ef98d55c073" containerName="main" containerID="cri-o://5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb" gracePeriod=60
Jul 13 19:52:56.449699 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.449642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:52:56.503614 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.503546    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:52:56.506172 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.506145    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:52:56.626348 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.626313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f71108-4328-480d-a2cb-3ef98d55c073-tls-certs\") pod \"21f71108-4328-480d-a2cb-3ef98d55c073\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") "
Jul 13 19:52:56.626549 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.626373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8rrk\" (UniqueName: \"kubernetes.io/projected/21f71108-4328-480d-a2cb-3ef98d55c073-kube-api-access-g8rrk\") pod \"21f71108-4328-480d-a2cb-3ef98d55c073\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") "
Jul 13 19:52:56.626549 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.626430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21f71108-4328-480d-a2cb-3ef98d55c073-tokenizer-uds\") pod \"21f71108-4328-480d-a2cb-3ef98d55c073\" (UID: \"21f71108-4328-480d-a2cb-3ef98d55c073\") "
Jul 13 19:52:56.626754 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.626729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f71108-4328-480d-a2cb-3ef98d55c073-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "21f71108-4328-480d-a2cb-3ef98d55c073" (UID: "21f71108-4328-480d-a2cb-3ef98d55c073"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:52:56.628685 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.628654    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f71108-4328-480d-a2cb-3ef98d55c073-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21f71108-4328-480d-a2cb-3ef98d55c073" (UID: "21f71108-4328-480d-a2cb-3ef98d55c073"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:52:56.628816 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.628704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f71108-4328-480d-a2cb-3ef98d55c073-kube-api-access-g8rrk" (OuterVolumeSpecName: "kube-api-access-g8rrk") pod "21f71108-4328-480d-a2cb-3ef98d55c073" (UID: "21f71108-4328-480d-a2cb-3ef98d55c073"). InnerVolumeSpecName "kube-api-access-g8rrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:52:56.668401 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.668366    2581 generic.go:356] "Generic (PLEG): container finished" podID="21f71108-4328-480d-a2cb-3ef98d55c073" containerID="5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb" exitCode=0
Jul 13 19:52:56.668538 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.668406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" event={"ID":"21f71108-4328-480d-a2cb-3ef98d55c073","Type":"ContainerDied","Data":"5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb"}
Jul 13 19:52:56.668538 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.668416    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"
Jul 13 19:52:56.668538 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.668443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26" event={"ID":"21f71108-4328-480d-a2cb-3ef98d55c073","Type":"ContainerDied","Data":"db3520d7bdd0cb6901861c1c343ad4c773f853c26f07befd4e2ed170e95202ea"}
Jul 13 19:52:56.668538 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.668463    2581 scope.go:117] "RemoveContainer" containerID="5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb"
Jul 13 19:52:56.678416 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.678397    2581 scope.go:117] "RemoveContainer" containerID="5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb"
Jul 13 19:52:56.678679 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:52:56.678660    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb\": container with ID starting with 5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb not found: ID does not exist" containerID="5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb"
Jul 13 19:52:56.678741 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.678692    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb"} err="failed to get container status \"5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb\": rpc error: code = NotFound desc = could not find container \"5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb\": container with ID starting with 5768ee4db18f2ce75352a76e70fe07a31a8ecdcda8a28a7b4e342d1ab01c4fcb not found: ID does not exist"
Jul 13 19:52:56.689382 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.689358    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"]
Jul 13 19:52:56.692976 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.692951    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26"]
Jul 13 19:52:56.728068 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.728035    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8rrk\" (UniqueName: \"kubernetes.io/projected/21f71108-4328-480d-a2cb-3ef98d55c073-kube-api-access-g8rrk\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:52:56.728068 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.728066    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21f71108-4328-480d-a2cb-3ef98d55c073-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:52:56.728250 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:56.728080    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f71108-4328-480d-a2cb-3ef98d55c073-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:52:57.605622 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:52:57.605586    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f71108-4328-480d-a2cb-3ef98d55c073" path="/var/lib/kubelet/pods/21f71108-4328-480d-a2cb-3ef98d55c073/volumes"
Jul 13 19:53:44.425167 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.425136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:53:44.808383 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.808351    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"]
Jul 13 19:53:44.808682 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.808668    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f71108-4328-480d-a2cb-3ef98d55c073" containerName="main"
Jul 13 19:53:44.808758 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.808683    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f71108-4328-480d-a2cb-3ef98d55c073" containerName="main"
Jul 13 19:53:44.808758 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.808747    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f71108-4328-480d-a2cb-3ef98d55c073" containerName="main"
Jul 13 19:53:44.811595 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.811574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:44.814987 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.814962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 13 19:53:44.815126 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.814995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-kwkw8\""
Jul 13 19:53:44.821055 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.821034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"]
Jul 13 19:53:44.844552 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.844521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:53:44.935119 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.935085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5565\" (UniqueName: \"kubernetes.io/projected/b0c7b365-0427-4fba-b2a6-75830563e83f-kube-api-access-q5565\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:44.935295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.935132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7b365-0427-4fba-b2a6-75830563e83f-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:44.935295 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:44.935211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0c7b365-0427-4fba-b2a6-75830563e83f-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.035781 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.035742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7b365-0427-4fba-b2a6-75830563e83f-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.035781 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.035797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0c7b365-0427-4fba-b2a6-75830563e83f-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.036028 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.035873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5565\" (UniqueName: \"kubernetes.io/projected/b0c7b365-0427-4fba-b2a6-75830563e83f-kube-api-access-q5565\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.036288 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.036270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0c7b365-0427-4fba-b2a6-75830563e83f-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.038410 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.038387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7b365-0427-4fba-b2a6-75830563e83f-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.045366 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.045335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5565\" (UniqueName: \"kubernetes.io/projected/b0c7b365-0427-4fba-b2a6-75830563e83f-kube-api-access-q5565\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.123278 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.123187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.259910 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.259885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"]
Jul 13 19:53:45.262395 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:53:45.262356    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c7b365_0427_4fba_b2a6_75830563e83f.slice/crio-5063632e0589b0951062b898a51d6d18177110b570df26b75a8b70f4d95fabae WatchSource:0}: Error finding container 5063632e0589b0951062b898a51d6d18177110b570df26b75a8b70f4d95fabae: Status 404 returned error can't find the container with id 5063632e0589b0951062b898a51d6d18177110b570df26b75a8b70f4d95fabae
Jul 13 19:53:45.906399 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.906359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" event={"ID":"b0c7b365-0427-4fba-b2a6-75830563e83f","Type":"ContainerStarted","Data":"38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5"}
Jul 13 19:53:45.906399 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.906404    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" event={"ID":"b0c7b365-0427-4fba-b2a6-75830563e83f","Type":"ContainerStarted","Data":"5063632e0589b0951062b898a51d6d18177110b570df26b75a8b70f4d95fabae"}
Jul 13 19:53:45.906948 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.906493    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:53:45.922475 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:53:45.922425    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" podStartSLOduration=1.92240787 podStartE2EDuration="1.92240787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:53:45.921587485 +0000 UTC m=+3862.821812880" watchObservedRunningTime="2026-07-13 19:53:45.92240787 +0000 UTC m=+3862.822633263"
Jul 13 19:54:10.776464 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:54:10.776428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:54:16.913349 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:54:16.913319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:54:23.880504 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:54:23.880475    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:54:23.892596 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:54:23.892575    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:54:27.804031 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:54:27.803968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:54:31.529187 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:54:31.529148    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 19:54:32.134007 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:54:32.133976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:55:53.776768 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:53.776734    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"]
Jul 13 19:55:53.785592 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:55:53.785552    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:55:53.785592 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:55:53.785592 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:55:53.785592 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5" cmd=["/bin/sleep","15"]
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:53.785614    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" message="" err=<
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:55:53.785630    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" podUID="b0c7b365-0427-4fba-b2a6-75830563e83f" containerName="main" containerID="cri-o://38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5"
Jul 13 19:55:53.785933 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:53.785686    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" podUID="b0c7b365-0427-4fba-b2a6-75830563e83f" containerName="main" containerID="cri-o://38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5" gracePeriod=60
Jul 13 19:55:54.047705 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.047676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:55:54.151015 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.150982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7b365-0427-4fba-b2a6-75830563e83f-tls-certs\") pod \"b0c7b365-0427-4fba-b2a6-75830563e83f\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") "
Jul 13 19:55:54.151186 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.151025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5565\" (UniqueName: \"kubernetes.io/projected/b0c7b365-0427-4fba-b2a6-75830563e83f-kube-api-access-q5565\") pod \"b0c7b365-0427-4fba-b2a6-75830563e83f\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") "
Jul 13 19:55:54.151186 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.151062    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0c7b365-0427-4fba-b2a6-75830563e83f-tokenizer-uds\") pod \"b0c7b365-0427-4fba-b2a6-75830563e83f\" (UID: \"b0c7b365-0427-4fba-b2a6-75830563e83f\") "
Jul 13 19:55:54.151436 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.151410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7b365-0427-4fba-b2a6-75830563e83f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b0c7b365-0427-4fba-b2a6-75830563e83f" (UID: "b0c7b365-0427-4fba-b2a6-75830563e83f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:55:54.153152 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.153129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c7b365-0427-4fba-b2a6-75830563e83f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0c7b365-0427-4fba-b2a6-75830563e83f" (UID: "b0c7b365-0427-4fba-b2a6-75830563e83f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:55:54.153250 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.153233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c7b365-0427-4fba-b2a6-75830563e83f-kube-api-access-q5565" (OuterVolumeSpecName: "kube-api-access-q5565") pod "b0c7b365-0427-4fba-b2a6-75830563e83f" (UID: "b0c7b365-0427-4fba-b2a6-75830563e83f"). InnerVolumeSpecName "kube-api-access-q5565". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:55:54.252458 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.252427    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0c7b365-0427-4fba-b2a6-75830563e83f-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:55:54.252458 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.252453    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7b365-0427-4fba-b2a6-75830563e83f-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:55:54.252458 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.252463    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5565\" (UniqueName: \"kubernetes.io/projected/b0c7b365-0427-4fba-b2a6-75830563e83f-kube-api-access-q5565\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:55:54.338233 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.338158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:55:54.530822 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.530775    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0c7b365-0427-4fba-b2a6-75830563e83f" containerID="38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5" exitCode=0
Jul 13 19:55:54.530997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.530836    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"
Jul 13 19:55:54.530997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.530858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" event={"ID":"b0c7b365-0427-4fba-b2a6-75830563e83f","Type":"ContainerDied","Data":"38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5"}
Jul 13 19:55:54.530997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.530928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4" event={"ID":"b0c7b365-0427-4fba-b2a6-75830563e83f","Type":"ContainerDied","Data":"5063632e0589b0951062b898a51d6d18177110b570df26b75a8b70f4d95fabae"}
Jul 13 19:55:54.530997 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.530949    2581 scope.go:117] "RemoveContainer" containerID="38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5"
Jul 13 19:55:54.540590 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.540573    2581 scope.go:117] "RemoveContainer" containerID="38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5"
Jul 13 19:55:54.540872 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:55:54.540853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5\": container with ID starting with 38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5 not found: ID does not exist" containerID="38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5"
Jul 13 19:55:54.540942 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.540880    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5"} err="failed to get container status \"38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5\": rpc error: code = NotFound desc = could not find container \"38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5\": container with ID starting with 38de2a4a07645f25edff71217660f901fbb96d5af918121d49eff7a9670e64f5 not found: ID does not exist"
Jul 13 19:55:54.550543 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.550523    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"]
Jul 13 19:55:54.554344 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:54.554312    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4"]
Jul 13 19:55:55.604617 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:55.604581    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c7b365-0427-4fba-b2a6-75830563e83f" path="/var/lib/kubelet/pods/b0c7b365-0427-4fba-b2a6-75830563e83f/volumes"
Jul 13 19:55:55.924834 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:55:55.924690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:56:20.468281 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:20.468209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:56:35.134685 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:56:35.134643    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 19:56:35.730799 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:35.730754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:56:58.310929 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.310838    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:56:58.601431 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.601333    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"]
Jul 13 19:56:58.601716 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.601700    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c7b365-0427-4fba-b2a6-75830563e83f" containerName="main"
Jul 13 19:56:58.601776 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.601719    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c7b365-0427-4fba-b2a6-75830563e83f" containerName="main"
Jul 13 19:56:58.601776 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.601772    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c7b365-0427-4fba-b2a6-75830563e83f" containerName="main"
Jul 13 19:56:58.604586 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.604569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.607911 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.607887    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 13 19:56:58.607993 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.607907    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-nzhfw\""
Jul 13 19:56:58.615909 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.615874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"]
Jul 13 19:56:58.625056 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.625036    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:56:58.703831 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.703800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.704036 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.703895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.704036 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.703927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc9w6\" (UniqueName: \"kubernetes.io/projected/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-kube-api-access-zc9w6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.805174 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.805134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.805379 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.805230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.805379 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.805273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc9w6\" (UniqueName: \"kubernetes.io/projected/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-kube-api-access-zc9w6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.805674 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.805653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.807740 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.807720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.812929 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.812906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc9w6\" (UniqueName: \"kubernetes.io/projected/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-kube-api-access-zc9w6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:58.913975 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:58.913880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:59.056175 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:59.056155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"]
Jul 13 19:56:59.058325 ip-10-0-131-203 kubenswrapper[2581]: W0713 19:56:59.058293    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5af6dc_42fd_420a_aaf0_16c90b26f1f8.slice/crio-4bfe0361ea35d50b1abf33247f7b96d7688affbb02efc5693189a47d5b1aaff3 WatchSource:0}: Error finding container 4bfe0361ea35d50b1abf33247f7b96d7688affbb02efc5693189a47d5b1aaff3: Status 404 returned error can't find the container with id 4bfe0361ea35d50b1abf33247f7b96d7688affbb02efc5693189a47d5b1aaff3
Jul 13 19:56:59.843576 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:59.843545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" event={"ID":"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8","Type":"ContainerStarted","Data":"28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1"}
Jul 13 19:56:59.844031 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:59.843591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" event={"ID":"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8","Type":"ContainerStarted","Data":"4bfe0361ea35d50b1abf33247f7b96d7688affbb02efc5693189a47d5b1aaff3"}
Jul 13 19:56:59.844031 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:59.843613    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:56:59.861391 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:56:59.861344    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" podStartSLOduration=1.861329641 podStartE2EDuration="1.861329641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:56:59.859497641 +0000 UTC m=+4056.759723033" watchObservedRunningTime="2026-07-13 19:56:59.861329641 +0000 UTC m=+4056.761555032"
Jul 13 19:57:30.849408 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:57:30.849380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:57:44.290532 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:57:44.290498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:57:44.352134 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:57:44.352097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:57:49.418290 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:57:49.418258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:58:38.731824 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:58:38.731708    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 19:58:39.327583 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:58:39.327550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 19:59:08.392865 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.392817    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"]
Jul 13 19:59:08.397053 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.397012    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:59:08.403253 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:59:08.403222    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:59:08.403253 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:59:08.403253 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:59:08.403253 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1" cmd=["/bin/sleep","15"]
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.403268    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" message="" err=<
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:59:08.403280    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" podUID="2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" containerName="main" containerID="cri-o://28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1"
Jul 13 19:59:08.403586 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.403317    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" podUID="2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" containerName="main" containerID="cri-o://28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1" gracePeriod=60
Jul 13 19:59:08.459121 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.459094    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:59:08.656616 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.656566    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:59:08.815386 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.815347    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tokenizer-uds\") pod \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") "
Jul 13 19:59:08.815386 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.815391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tls-certs\") pod \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") "
Jul 13 19:59:08.815688 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.815496    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc9w6\" (UniqueName: \"kubernetes.io/projected/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-kube-api-access-zc9w6\") pod \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\" (UID: \"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8\") "
Jul 13 19:59:08.815688 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.815664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" (UID: "2c5af6dc-42fd-420a-aaf0-16c90b26f1f8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:59:08.815836 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.815775    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:59:08.817631 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.817605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-kube-api-access-zc9w6" (OuterVolumeSpecName: "kube-api-access-zc9w6") pod "2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" (UID: "2c5af6dc-42fd-420a-aaf0-16c90b26f1f8"). InnerVolumeSpecName "kube-api-access-zc9w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:59:08.817631 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.817620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" (UID: "2c5af6dc-42fd-420a-aaf0-16c90b26f1f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:59:08.916624 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.916532    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:59:08.916624 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:08.916560    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc9w6\" (UniqueName: \"kubernetes.io/projected/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8-kube-api-access-zc9w6\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 19:59:09.477765 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.477730    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" containerID="28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1" exitCode=0
Jul 13 19:59:09.478376 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.477776    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"
Jul 13 19:59:09.478376 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.477826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" event={"ID":"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8","Type":"ContainerDied","Data":"28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1"}
Jul 13 19:59:09.478376 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.477921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs" event={"ID":"2c5af6dc-42fd-420a-aaf0-16c90b26f1f8","Type":"ContainerDied","Data":"4bfe0361ea35d50b1abf33247f7b96d7688affbb02efc5693189a47d5b1aaff3"}
Jul 13 19:59:09.478376 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.477951    2581 scope.go:117] "RemoveContainer" containerID="28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1"
Jul 13 19:59:09.487940 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.487919    2581 scope.go:117] "RemoveContainer" containerID="28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1"
Jul 13 19:59:09.488204 ip-10-0-131-203 kubenswrapper[2581]: E0713 19:59:09.488182    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1\": container with ID starting with 28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1 not found: ID does not exist" containerID="28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1"
Jul 13 19:59:09.488276 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.488215    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1"} err="failed to get container status \"28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1\": rpc error: code = NotFound desc = could not find container \"28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1\": container with ID starting with 28902fcdc31ead17f6130b029682ab5cac93221015781c47bf2a7ce78a13e0a1 not found: ID does not exist"
Jul 13 19:59:09.499540 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.499511    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"]
Jul 13 19:59:09.503331 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.503311    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs"]
Jul 13 19:59:09.603298 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:09.603265    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" path="/var/lib/kubelet/pods/2c5af6dc-42fd-420a-aaf0-16c90b26f1f8/volumes"
Jul 13 19:59:23.910699 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:23.910595    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:59:23.923921 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:23.923897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 19:59:37.681747 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:37.681708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:59:37.739890 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:37.739832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 19:59:51.911747 ip-10-0-131-203 kubenswrapper[2581]: I0713 19:59:51.911672    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:00:10.291948 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.291915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:00:10.434436 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.434395    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"]
Jul 13 20:00:10.434843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.434807    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" containerName="main"
Jul 13 20:00:10.434843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.434842    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" containerName="main"
Jul 13 20:00:10.435051 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.434936    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5af6dc-42fd-420a-aaf0-16c90b26f1f8" containerName="main"
Jul 13 20:00:10.444082 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.444061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.447016 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.446958    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-25hgf\""
Jul 13 20:00:10.447255 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.447235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 13 20:00:10.448530 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.448509    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"]
Jul 13 20:00:10.541138 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.541089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c87c395-4aea-4405-921a-49a72ce22e98-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.541315 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.541157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c87c395-4aea-4405-921a-49a72ce22e98-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.541315 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.541270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwxt\" (UniqueName: \"kubernetes.io/projected/9c87c395-4aea-4405-921a-49a72ce22e98-kube-api-access-mpwxt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.642447 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.642339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c87c395-4aea-4405-921a-49a72ce22e98-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.642447 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.642405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c87c395-4aea-4405-921a-49a72ce22e98-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.642672 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.642491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwxt\" (UniqueName: \"kubernetes.io/projected/9c87c395-4aea-4405-921a-49a72ce22e98-kube-api-access-mpwxt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.642741 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.642722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c87c395-4aea-4405-921a-49a72ce22e98-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.645275 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.645250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c87c395-4aea-4405-921a-49a72ce22e98-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.653584 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.653555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwxt\" (UniqueName: \"kubernetes.io/projected/9c87c395-4aea-4405-921a-49a72ce22e98-kube-api-access-mpwxt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.706973 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.706937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:00:10.754961 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.754927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:10.888989 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:10.888965    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"]
Jul 13 20:00:10.890902 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:00:10.890872    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c87c395_4aea_4405_921a_49a72ce22e98.slice/crio-060e7b2111bf7dda0e993d6cf04c9dbd525bfcedbda32e8a1f4758fc29cfa819 WatchSource:0}: Error finding container 060e7b2111bf7dda0e993d6cf04c9dbd525bfcedbda32e8a1f4758fc29cfa819: Status 404 returned error can't find the container with id 060e7b2111bf7dda0e993d6cf04c9dbd525bfcedbda32e8a1f4758fc29cfa819
Jul 13 20:00:11.788371 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:11.788314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" event={"ID":"9c87c395-4aea-4405-921a-49a72ce22e98","Type":"ContainerStarted","Data":"870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c"}
Jul 13 20:00:11.788834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:11.788385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" event={"ID":"9c87c395-4aea-4405-921a-49a72ce22e98","Type":"ContainerStarted","Data":"060e7b2111bf7dda0e993d6cf04c9dbd525bfcedbda32e8a1f4758fc29cfa819"}
Jul 13 20:00:11.788834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:11.788510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:11.805088 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:11.805037    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" podStartSLOduration=1.80502272 podStartE2EDuration="1.80502272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:00:11.803574381 +0000 UTC m=+4248.703799773" watchObservedRunningTime="2026-07-13 20:00:11.80502272 +0000 UTC m=+4248.705248111"
Jul 13 20:00:18.296277 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:18.296247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:00:42.328456 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:00:42.328397    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:00:42.795056 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:42.795027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:00:42.938286 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:00:42.938254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:01:06.885300 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:06.885266    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:01:49.649943 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:49.649899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:01:49.650829 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:49.650773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"]
Jul 13 20:01:49.661009 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:01:49.660975    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:01:49.661009 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:01:49.661009 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:01:49.661009 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c" cmd=["/bin/sleep","15"]
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:49.661026    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" message="" err=<
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:01:49.661044    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" podUID="9c87c395-4aea-4405-921a-49a72ce22e98" containerName="main" containerID="cri-o://870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c"
Jul 13 20:01:49.661342 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:49.661099    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" podUID="9c87c395-4aea-4405-921a-49a72ce22e98" containerName="main" containerID="cri-o://870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c" gracePeriod=60
Jul 13 20:01:49.713178 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:49.713144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:01:49.914526 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:49.914501    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:01:50.005748 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.005711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c87c395-4aea-4405-921a-49a72ce22e98-tokenizer-uds\") pod \"9c87c395-4aea-4405-921a-49a72ce22e98\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") "
Jul 13 20:01:50.005958 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.005774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c87c395-4aea-4405-921a-49a72ce22e98-tls-certs\") pod \"9c87c395-4aea-4405-921a-49a72ce22e98\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") "
Jul 13 20:01:50.005958 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.005862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpwxt\" (UniqueName: \"kubernetes.io/projected/9c87c395-4aea-4405-921a-49a72ce22e98-kube-api-access-mpwxt\") pod \"9c87c395-4aea-4405-921a-49a72ce22e98\" (UID: \"9c87c395-4aea-4405-921a-49a72ce22e98\") "
Jul 13 20:01:50.006079 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.006029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c87c395-4aea-4405-921a-49a72ce22e98-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9c87c395-4aea-4405-921a-49a72ce22e98" (UID: "9c87c395-4aea-4405-921a-49a72ce22e98"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:01:50.006147 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.006123    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c87c395-4aea-4405-921a-49a72ce22e98-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:01:50.008037 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.008008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c87c395-4aea-4405-921a-49a72ce22e98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c87c395-4aea-4405-921a-49a72ce22e98" (UID: "9c87c395-4aea-4405-921a-49a72ce22e98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:01:50.008171 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.008061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c87c395-4aea-4405-921a-49a72ce22e98-kube-api-access-mpwxt" (OuterVolumeSpecName: "kube-api-access-mpwxt") pod "9c87c395-4aea-4405-921a-49a72ce22e98" (UID: "9c87c395-4aea-4405-921a-49a72ce22e98"). InnerVolumeSpecName "kube-api-access-mpwxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:01:50.106850 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.106815    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpwxt\" (UniqueName: \"kubernetes.io/projected/9c87c395-4aea-4405-921a-49a72ce22e98-kube-api-access-mpwxt\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:01:50.106850 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.106845    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c87c395-4aea-4405-921a-49a72ce22e98-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:01:50.269567 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.269533    2581 generic.go:356] "Generic (PLEG): container finished" podID="9c87c395-4aea-4405-921a-49a72ce22e98" containerID="870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c" exitCode=0
Jul 13 20:01:50.269777 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.269576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" event={"ID":"9c87c395-4aea-4405-921a-49a72ce22e98","Type":"ContainerDied","Data":"870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c"}
Jul 13 20:01:50.269777 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.269616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s" event={"ID":"9c87c395-4aea-4405-921a-49a72ce22e98","Type":"ContainerDied","Data":"060e7b2111bf7dda0e993d6cf04c9dbd525bfcedbda32e8a1f4758fc29cfa819"}
Jul 13 20:01:50.269777 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.269635    2581 scope.go:117] "RemoveContainer" containerID="870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c"
Jul 13 20:01:50.269777 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.269581    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"
Jul 13 20:01:50.279454 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.279436    2581 scope.go:117] "RemoveContainer" containerID="870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c"
Jul 13 20:01:50.279703 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:01:50.279684    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c\": container with ID starting with 870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c not found: ID does not exist" containerID="870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c"
Jul 13 20:01:50.279748 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.279712    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c"} err="failed to get container status \"870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c\": rpc error: code = NotFound desc = could not find container \"870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c\": container with ID starting with 870072a88179df76e42ebdb112029b8f41a97cf57a5ec03a27a17973155cb44c not found: ID does not exist"
Jul 13 20:01:50.297031 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.296992    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"]
Jul 13 20:01:50.297767 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:50.297746    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s"]
Jul 13 20:01:51.603893 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:01:51.603857    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c87c395-4aea-4405-921a-49a72ce22e98" path="/var/lib/kubelet/pods/9c87c395-4aea-4405-921a-49a72ce22e98/volumes"
Jul 13 20:02:23.914527 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:23.914489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:02:24.165636 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.165550    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"]
Jul 13 20:02:24.165903 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.165889    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c87c395-4aea-4405-921a-49a72ce22e98" containerName="main"
Jul 13 20:02:24.165903 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.165904    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c87c395-4aea-4405-921a-49a72ce22e98" containerName="main"
Jul 13 20:02:24.166014 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.165953    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c87c395-4aea-4405-921a-49a72ce22e98" containerName="main"
Jul 13 20:02:24.168961 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.168941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.172115 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.172089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-fvf8k\""
Jul 13 20:02:24.172267 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.172246    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 13 20:02:24.190897 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.190875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"]
Jul 13 20:02:24.198661 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.198631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.198801 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.198701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.198801 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.198747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5492c87-cc6f-453c-82d8-9d7e14f17284-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.198899 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.198782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.198899 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.198858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb94m\" (UniqueName: \"kubernetes.io/projected/d5492c87-cc6f-453c-82d8-9d7e14f17284-kube-api-access-fb94m\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.227254 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.227225    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:02:24.299966 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.299933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb94m\" (UniqueName: \"kubernetes.io/projected/d5492c87-cc6f-453c-82d8-9d7e14f17284-kube-api-access-fb94m\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.300170 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.299992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.300170 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.300030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.300170 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.300060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5492c87-cc6f-453c-82d8-9d7e14f17284-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.300170 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.300110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.300523 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.300500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.300523 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.300516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.300629 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.300548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.302555 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.302535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5492c87-cc6f-453c-82d8-9d7e14f17284-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.308455 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.308430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb94m\" (UniqueName: \"kubernetes.io/projected/d5492c87-cc6f-453c-82d8-9d7e14f17284-kube-api-access-fb94m\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567bmx62\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.479447 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.479412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:24.615844 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:24.615820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"]
Jul 13 20:02:24.618009 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:02:24.617973    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5492c87_cc6f_453c_82d8_9d7e14f17284.slice/crio-6babec8afb099123f4b084045d260647c4f716385345d7eb6e6ed4e986652bb1 WatchSource:0}: Error finding container 6babec8afb099123f4b084045d260647c4f716385345d7eb6e6ed4e986652bb1: Status 404 returned error can't find the container with id 6babec8afb099123f4b084045d260647c4f716385345d7eb6e6ed4e986652bb1
Jul 13 20:02:25.446645 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:25.446546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" event={"ID":"d5492c87-cc6f-453c-82d8-9d7e14f17284","Type":"ContainerStarted","Data":"eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da"}
Jul 13 20:02:25.446645 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:25.446643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" event={"ID":"d5492c87-cc6f-453c-82d8-9d7e14f17284","Type":"ContainerStarted","Data":"6babec8afb099123f4b084045d260647c4f716385345d7eb6e6ed4e986652bb1"}
Jul 13 20:02:25.447093 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:25.446670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:02:25.464448 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:25.464398    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" podStartSLOduration=1.464384293 podStartE2EDuration="1.464384293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:02:25.462438556 +0000 UTC m=+4382.362663947" watchObservedRunningTime="2026-07-13 20:02:25.464384293 +0000 UTC m=+4382.364609684"
Jul 13 20:02:45.939609 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:02:45.939515    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:02:46.549424 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:46.549393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:02:56.452956 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:02:56.452929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:03:00.441950 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:03:00.441922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:03:24.570360 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:03:24.570284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:04:05.908726 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:05.908695    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"]
Jul 13 20:04:05.912262 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:05.912242    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:05.914163 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:05.914138    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:04:05.914163 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:05.914138    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 13 20:04:05.922234 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:05.922214    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"]
Jul 13 20:04:05.930146 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:05.930126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:04:06.074137 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.074104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11057863-d56c-4768-80cb-3042cbfe1bd7-tls-certs\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.074307 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.074189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-home\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.074307 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.074212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-tmp-dir\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.074307 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.074230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-dshm\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.074307 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.074291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhncz\" (UniqueName: \"kubernetes.io/projected/11057863-d56c-4768-80cb-3042cbfe1bd7-kube-api-access-vhncz\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.074467 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.074312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-kserve-provision-location\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.074467 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.074340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-model-cache\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.128091 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.128063    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"]
Jul 13 20:04:06.131483 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.131461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.133356 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.133333    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-gncl2\""
Jul 13 20:04:06.142986 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.142966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"]
Jul 13 20:04:06.163146 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.163058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:04:06.175129 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhncz\" (UniqueName: \"kubernetes.io/projected/11057863-d56c-4768-80cb-3042cbfe1bd7-kube-api-access-vhncz\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175240 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-kserve-provision-location\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175240 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-model-cache\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175353 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11057863-d56c-4768-80cb-3042cbfe1bd7-tls-certs\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175468 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-home\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175531 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-tmp-dir\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175531 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-model-cache\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175531 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-dshm\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175690 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-kserve-provision-location\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175749 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-home\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.175874 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.175853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-tmp-dir\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.177680 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.177653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-dshm\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.177957 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.177941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11057863-d56c-4768-80cb-3042cbfe1bd7-tls-certs\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.183542 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.183524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhncz\" (UniqueName: \"kubernetes.io/projected/11057863-d56c-4768-80cb-3042cbfe1bd7-kube-api-access-vhncz\") pod \"stop-feature-test-kserve-5f875cbf68-7df7h\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.223512 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.223485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:06.276842 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.276748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.276965 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.276883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtb6\" (UniqueName: \"kubernetes.io/projected/834c278a-adc6-494c-833f-9ed1e843dcb1-kube-api-access-vbtb6\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.277042 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.276996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.277151 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.277127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834c278a-adc6-494c-833f-9ed1e843dcb1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.362599 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.362568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"]
Jul 13 20:04:06.365104 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:04:06.365076    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11057863_d56c_4768_80cb_3042cbfe1bd7.slice/crio-1e9b75515dab277afb8f96a11a97041dd1e55cd35ff47496a5667be1de59c3fb WatchSource:0}: Error finding container 1e9b75515dab277afb8f96a11a97041dd1e55cd35ff47496a5667be1de59c3fb: Status 404 returned error can't find the container with id 1e9b75515dab277afb8f96a11a97041dd1e55cd35ff47496a5667be1de59c3fb
Jul 13 20:04:06.377882 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.377862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtb6\" (UniqueName: \"kubernetes.io/projected/834c278a-adc6-494c-833f-9ed1e843dcb1-kube-api-access-vbtb6\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.377969 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.377911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.378150 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.377990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834c278a-adc6-494c-833f-9ed1e843dcb1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.378150 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.378037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.378304 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.378281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.378372 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.378311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.380573 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.380552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834c278a-adc6-494c-833f-9ed1e843dcb1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.386542 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.386520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtb6\" (UniqueName: \"kubernetes.io/projected/834c278a-adc6-494c-833f-9ed1e843dcb1-kube-api-access-vbtb6\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.444732 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.444695    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:06.594708 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.594484    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"]
Jul 13 20:04:06.596964 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:04:06.596931    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834c278a_adc6_494c_833f_9ed1e843dcb1.slice/crio-0963085478d159191bd86ce488997e33a9f75ac464fd998bf61f78ec097ba482 WatchSource:0}: Error finding container 0963085478d159191bd86ce488997e33a9f75ac464fd998bf61f78ec097ba482: Status 404 returned error can't find the container with id 0963085478d159191bd86ce488997e33a9f75ac464fd998bf61f78ec097ba482
Jul 13 20:04:06.937670 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.937622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" event={"ID":"834c278a-adc6-494c-833f-9ed1e843dcb1","Type":"ContainerStarted","Data":"dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26"}
Jul 13 20:04:06.938180 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.937695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" event={"ID":"834c278a-adc6-494c-833f-9ed1e843dcb1","Type":"ContainerStarted","Data":"0963085478d159191bd86ce488997e33a9f75ac464fd998bf61f78ec097ba482"}
Jul 13 20:04:06.939325 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.939296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" event={"ID":"11057863-d56c-4768-80cb-3042cbfe1bd7","Type":"ContainerStarted","Data":"e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5"}
Jul 13 20:04:06.939472 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:06.939355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" event={"ID":"11057863-d56c-4768-80cb-3042cbfe1bd7","Type":"ContainerStarted","Data":"1e9b75515dab277afb8f96a11a97041dd1e55cd35ff47496a5667be1de59c3fb"}
Jul 13 20:04:07.946609 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:07.946574    2581 generic.go:356] "Generic (PLEG): container finished" podID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerID="dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26" exitCode=0
Jul 13 20:04:07.947134 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:07.946657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" event={"ID":"834c278a-adc6-494c-833f-9ed1e843dcb1","Type":"ContainerDied","Data":"dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26"}
Jul 13 20:04:08.953840 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:08.953784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" event={"ID":"834c278a-adc6-494c-833f-9ed1e843dcb1","Type":"ContainerStarted","Data":"fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7"}
Jul 13 20:04:08.954217 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:08.953907    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:08.972618 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:08.972575    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" podStartSLOduration=2.181627431 podStartE2EDuration="2.972561354s" totalImagesPullingTime="0s" totalInitContainerRuntime="790.933923ms" isStatefulPod=false podCreationTimestamp="2026-07-13 20:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:04:08.970215889 +0000 UTC m=+4485.870441280" watchObservedRunningTime="2026-07-13 20:04:08.972561354 +0000 UTC m=+4485.872786745"
Jul 13 20:04:09.960010 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:09.959980    2581 generic.go:356] "Generic (PLEG): container finished" podID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerID="e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5" exitCode=0
Jul 13 20:04:09.960417 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:09.960056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" event={"ID":"11057863-d56c-4768-80cb-3042cbfe1bd7","Type":"ContainerDied","Data":"e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5"}
Jul 13 20:04:10.968218 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:10.968179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" event={"ID":"11057863-d56c-4768-80cb-3042cbfe1bd7","Type":"ContainerStarted","Data":"78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5"}
Jul 13 20:04:10.994240 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:10.994159    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podStartSLOduration=2.777957865 podStartE2EDuration="5.994139278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.216181413s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:04:10.987842094 +0000 UTC m=+4487.888067507" watchObservedRunningTime="2026-07-13 20:04:10.994139278 +0000 UTC m=+4487.894364671"
Jul 13 20:04:11.223852 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:11.223719    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:16.224052 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:16.224017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:04:16.225707 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:16.225682    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:04:23.947200 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:23.947087    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:04:23.960260 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:23.960237    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:04:26.224541 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:26.224487    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:04:32.111737 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.111689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:04:32.113847 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.113815    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"]
Jul 13 20:04:32.123262 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:04:32.123225    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:04:32.123262 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:04:32.123262 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:04:32.123262 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da" cmd=["/bin/sleep","15"]
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.123277    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" message="" err=<
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:04:32.123289    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" podUID="d5492c87-cc6f-453c-82d8-9d7e14f17284" containerName="main" containerID="cri-o://eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da"
Jul 13 20:04:32.123529 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.123344    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" podUID="d5492c87-cc6f-453c-82d8-9d7e14f17284" containerName="main" containerID="cri-o://eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da" gracePeriod=60
Jul 13 20:04:32.207073 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.207043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:04:32.375937 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.375911    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:04:32.522181 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5492c87-cc6f-453c-82d8-9d7e14f17284-tls-certs\") pod \"d5492c87-cc6f-453c-82d8-9d7e14f17284\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") "
Jul 13 20:04:32.522384 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522201    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-cachi2-cache\") pod \"d5492c87-cc6f-453c-82d8-9d7e14f17284\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") "
Jul 13 20:04:32.522384 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-uds\") pod \"d5492c87-cc6f-453c-82d8-9d7e14f17284\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") "
Jul 13 20:04:32.522384 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb94m\" (UniqueName: \"kubernetes.io/projected/d5492c87-cc6f-453c-82d8-9d7e14f17284-kube-api-access-fb94m\") pod \"d5492c87-cc6f-453c-82d8-9d7e14f17284\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") "
Jul 13 20:04:32.522384 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522293    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-cache\") pod \"d5492c87-cc6f-453c-82d8-9d7e14f17284\" (UID: \"d5492c87-cc6f-453c-82d8-9d7e14f17284\") "
Jul 13 20:04:32.522611 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522522    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d5492c87-cc6f-453c-82d8-9d7e14f17284" (UID: "d5492c87-cc6f-453c-82d8-9d7e14f17284"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:04:32.522611 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522540    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "d5492c87-cc6f-453c-82d8-9d7e14f17284" (UID: "d5492c87-cc6f-453c-82d8-9d7e14f17284"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:04:32.522695 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.522619    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d5492c87-cc6f-453c-82d8-9d7e14f17284" (UID: "d5492c87-cc6f-453c-82d8-9d7e14f17284"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:04:32.524469 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.524445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5492c87-cc6f-453c-82d8-9d7e14f17284-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5492c87-cc6f-453c-82d8-9d7e14f17284" (UID: "d5492c87-cc6f-453c-82d8-9d7e14f17284"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:04:32.524613 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.524587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5492c87-cc6f-453c-82d8-9d7e14f17284-kube-api-access-fb94m" (OuterVolumeSpecName: "kube-api-access-fb94m") pod "d5492c87-cc6f-453c-82d8-9d7e14f17284" (UID: "d5492c87-cc6f-453c-82d8-9d7e14f17284"). InnerVolumeSpecName "kube-api-access-fb94m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:04:32.623297 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.623216    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5492c87-cc6f-453c-82d8-9d7e14f17284-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:04:32.623297 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.623244    2581 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-cachi2-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:04:32.623297 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.623253    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:04:32.623297 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.623260    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb94m\" (UniqueName: \"kubernetes.io/projected/d5492c87-cc6f-453c-82d8-9d7e14f17284-kube-api-access-fb94m\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:04:32.623297 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:32.623270    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5492c87-cc6f-453c-82d8-9d7e14f17284-tokenizer-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:04:33.093468 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.093426    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5492c87-cc6f-453c-82d8-9d7e14f17284" containerID="eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da" exitCode=0
Jul 13 20:04:33.093673 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.093526    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"
Jul 13 20:04:33.093673 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.093554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" event={"ID":"d5492c87-cc6f-453c-82d8-9d7e14f17284","Type":"ContainerDied","Data":"eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da"}
Jul 13 20:04:33.093673 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.093600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62" event={"ID":"d5492c87-cc6f-453c-82d8-9d7e14f17284","Type":"ContainerDied","Data":"6babec8afb099123f4b084045d260647c4f716385345d7eb6e6ed4e986652bb1"}
Jul 13 20:04:33.093673 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.093621    2581 scope.go:117] "RemoveContainer" containerID="eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da"
Jul 13 20:04:33.104760 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.104730    2581 scope.go:117] "RemoveContainer" containerID="eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da"
Jul 13 20:04:33.105056 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:04:33.105039    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da\": container with ID starting with eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da not found: ID does not exist" containerID="eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da"
Jul 13 20:04:33.105127 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.105065    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da"} err="failed to get container status \"eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da\": rpc error: code = NotFound desc = could not find container \"eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da\": container with ID starting with eecfcd982c5c2f2c496a20a4872a085b716f81bfcd1adc41fe7cf7d6486667da not found: ID does not exist"
Jul 13 20:04:33.113756 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.113735    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"]
Jul 13 20:04:33.117767 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.117746    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62"]
Jul 13 20:04:33.611771 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:33.611742    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5492c87-cc6f-453c-82d8-9d7e14f17284" path="/var/lib/kubelet/pods/d5492c87-cc6f-453c-82d8-9d7e14f17284/volumes"
Jul 13 20:04:36.224406 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:36.224352    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:04:39.962351 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:39.962322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:04:46.224598 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:46.224561    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:04:49.550877 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:04:49.550825    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:04:50.185240 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:50.185209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:04:55.399282 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:55.399247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:04:56.224898 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:04:56.224856    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:05:06.224607 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:06.224564    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:05:16.224594 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:16.224545    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:05:21.986549 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:21.986509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:05:26.224898 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:26.224859    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:05:36.224340 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:36.224286    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:05:38.431490 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:38.431440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:05:40.122931 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:40.122902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:05:40.437691 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:40.437608    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:05:46.224570 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:46.224480    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:05:56.224266 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:05:56.224215    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 13 20:06:06.234249 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:06.234217    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:06:06.242498 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:06.242470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:06:26.237226 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:26.237177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:06:41.358932 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.358894    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"]
Jul 13 20:06:41.388695 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.388663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:06:41.456438 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.456404    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"]
Jul 13 20:06:41.466095 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:06:41.466055    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:06:41.466095 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:06:41.466095 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:06:41.466095 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7" cmd=["/bin/sleep","15"]
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.466110    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" message="" err=<
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:06:41.466128    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerName="main" containerID="cri-o://fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7"
Jul 13 20:06:41.466391 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.466181    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerName="main" containerID="cri-o://fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7" gracePeriod=60
Jul 13 20:06:41.574000 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.573968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:06:41.678175 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.678141    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:06:41.724565 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.724540    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:06:41.767602 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.767571    2581 generic.go:356] "Generic (PLEG): container finished" podID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerID="fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7" exitCode=0
Jul 13 20:06:41.767764 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.767615    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"
Jul 13 20:06:41.767764 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.767653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" event={"ID":"834c278a-adc6-494c-833f-9ed1e843dcb1","Type":"ContainerDied","Data":"fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7"}
Jul 13 20:06:41.767764 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.767715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj" event={"ID":"834c278a-adc6-494c-833f-9ed1e843dcb1","Type":"ContainerDied","Data":"0963085478d159191bd86ce488997e33a9f75ac464fd998bf61f78ec097ba482"}
Jul 13 20:06:41.767764 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.767747    2581 scope.go:117] "RemoveContainer" containerID="fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7"
Jul 13 20:06:41.777962 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.777942    2581 scope.go:117] "RemoveContainer" containerID="dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26"
Jul 13 20:06:41.785601 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.785586    2581 scope.go:117] "RemoveContainer" containerID="fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7"
Jul 13 20:06:41.785870 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:06:41.785848    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7\": container with ID starting with fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7 not found: ID does not exist" containerID="fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7"
Jul 13 20:06:41.785960 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.785880    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7"} err="failed to get container status \"fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7\": rpc error: code = NotFound desc = could not find container \"fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7\": container with ID starting with fc8a1686b15e0e538f4918a0300f9d27b524bb496be6f22fac333bbe9b2c1dc7 not found: ID does not exist"
Jul 13 20:06:41.785960 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.785903    2581 scope.go:117] "RemoveContainer" containerID="dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26"
Jul 13 20:06:41.786202 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:06:41.786183    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26\": container with ID starting with dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26 not found: ID does not exist" containerID="dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26"
Jul 13 20:06:41.786249 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.786207    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26"} err="failed to get container status \"dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26\": rpc error: code = NotFound desc = could not find container \"dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26\": container with ID starting with dafcbf14e93c71d2dbc0e79b284b8a28a84186cb7faf52a49b341fc9ab99ad26 not found: ID does not exist"
Jul 13 20:06:41.793463 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.793443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834c278a-adc6-494c-833f-9ed1e843dcb1-tls-certs\") pod \"834c278a-adc6-494c-833f-9ed1e843dcb1\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") "
Jul 13 20:06:41.793540 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.793484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-kserve-provision-location\") pod \"834c278a-adc6-494c-833f-9ed1e843dcb1\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") "
Jul 13 20:06:41.793540 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.793514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbtb6\" (UniqueName: \"kubernetes.io/projected/834c278a-adc6-494c-833f-9ed1e843dcb1-kube-api-access-vbtb6\") pod \"834c278a-adc6-494c-833f-9ed1e843dcb1\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") "
Jul 13 20:06:41.793619 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.793547    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-tokenizer-uds\") pod \"834c278a-adc6-494c-833f-9ed1e843dcb1\" (UID: \"834c278a-adc6-494c-833f-9ed1e843dcb1\") "
Jul 13 20:06:41.793937 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.793915    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "834c278a-adc6-494c-833f-9ed1e843dcb1" (UID: "834c278a-adc6-494c-833f-9ed1e843dcb1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:41.794375 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.794315    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "834c278a-adc6-494c-833f-9ed1e843dcb1" (UID: "834c278a-adc6-494c-833f-9ed1e843dcb1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:41.795470 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.795448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834c278a-adc6-494c-833f-9ed1e843dcb1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "834c278a-adc6-494c-833f-9ed1e843dcb1" (UID: "834c278a-adc6-494c-833f-9ed1e843dcb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:06:41.795702 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.795688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834c278a-adc6-494c-833f-9ed1e843dcb1-kube-api-access-vbtb6" (OuterVolumeSpecName: "kube-api-access-vbtb6") pod "834c278a-adc6-494c-833f-9ed1e843dcb1" (UID: "834c278a-adc6-494c-833f-9ed1e843dcb1"). InnerVolumeSpecName "kube-api-access-vbtb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:06:41.895116 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.895026    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834c278a-adc6-494c-833f-9ed1e843dcb1-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:41.895116 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.895064    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:41.895116 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.895079    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbtb6\" (UniqueName: \"kubernetes.io/projected/834c278a-adc6-494c-833f-9ed1e843dcb1-kube-api-access-vbtb6\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:41.895116 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:41.895093    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/834c278a-adc6-494c-833f-9ed1e843dcb1-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:42.097317 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:42.091845    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"]
Jul 13 20:06:42.100350 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:42.100321    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-75mhj"]
Jul 13 20:06:43.604634 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:43.604591    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" path="/var/lib/kubelet/pods/834c278a-adc6-494c-833f-9ed1e843dcb1/volumes"
Jul 13 20:06:53.186008 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:06:53.185954    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:06:53.826618 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:53.826587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:06:54.859576 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:54.859542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:06:54.922395 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:54.922342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:06:56.384099 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:56.384058    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main" containerID="cri-o://78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5" gracePeriod=45
Jul 13 20:06:57.137150 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.137123    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:06:57.239685 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.239655    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-kserve-provision-location\") pod \"11057863-d56c-4768-80cb-3042cbfe1bd7\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") "
Jul 13 20:06:57.239905 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.239701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-home\") pod \"11057863-d56c-4768-80cb-3042cbfe1bd7\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") "
Jul 13 20:06:57.239905 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.239728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-model-cache\") pod \"11057863-d56c-4768-80cb-3042cbfe1bd7\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") "
Jul 13 20:06:57.239905 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.239777    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-dshm\") pod \"11057863-d56c-4768-80cb-3042cbfe1bd7\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") "
Jul 13 20:06:57.239905 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.239826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhncz\" (UniqueName: \"kubernetes.io/projected/11057863-d56c-4768-80cb-3042cbfe1bd7-kube-api-access-vhncz\") pod \"11057863-d56c-4768-80cb-3042cbfe1bd7\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") "
Jul 13 20:06:57.239905 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.239892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11057863-d56c-4768-80cb-3042cbfe1bd7-tls-certs\") pod \"11057863-d56c-4768-80cb-3042cbfe1bd7\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") "
Jul 13 20:06:57.240192 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.239923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-tmp-dir\") pod \"11057863-d56c-4768-80cb-3042cbfe1bd7\" (UID: \"11057863-d56c-4768-80cb-3042cbfe1bd7\") "
Jul 13 20:06:57.240192 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.240066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "11057863-d56c-4768-80cb-3042cbfe1bd7" (UID: "11057863-d56c-4768-80cb-3042cbfe1bd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:57.240294 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.240276    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:57.240502 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.240478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-home" (OuterVolumeSpecName: "home") pod "11057863-d56c-4768-80cb-3042cbfe1bd7" (UID: "11057863-d56c-4768-80cb-3042cbfe1bd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:57.242269 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.242245    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-dshm" (OuterVolumeSpecName: "dshm") pod "11057863-d56c-4768-80cb-3042cbfe1bd7" (UID: "11057863-d56c-4768-80cb-3042cbfe1bd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:57.242663 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.242637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11057863-d56c-4768-80cb-3042cbfe1bd7-kube-api-access-vhncz" (OuterVolumeSpecName: "kube-api-access-vhncz") pod "11057863-d56c-4768-80cb-3042cbfe1bd7" (UID: "11057863-d56c-4768-80cb-3042cbfe1bd7"). InnerVolumeSpecName "kube-api-access-vhncz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:06:57.243257 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.243215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11057863-d56c-4768-80cb-3042cbfe1bd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11057863-d56c-4768-80cb-3042cbfe1bd7" (UID: "11057863-d56c-4768-80cb-3042cbfe1bd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:06:57.253272 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.253247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11057863-d56c-4768-80cb-3042cbfe1bd7" (UID: "11057863-d56c-4768-80cb-3042cbfe1bd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:57.303899 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.303864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11057863-d56c-4768-80cb-3042cbfe1bd7" (UID: "11057863-d56c-4768-80cb-3042cbfe1bd7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:57.341150 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.341115    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:57.341150 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.341143    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:57.341150 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.341156    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhncz\" (UniqueName: \"kubernetes.io/projected/11057863-d56c-4768-80cb-3042cbfe1bd7-kube-api-access-vhncz\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:57.341390 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.341168    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11057863-d56c-4768-80cb-3042cbfe1bd7-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:57.341390 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.341182    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:57.341390 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.341193    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11057863-d56c-4768-80cb-3042cbfe1bd7-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:06:57.850635 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.850601    2581 generic.go:356] "Generic (PLEG): container finished" podID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerID="78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5" exitCode=0
Jul 13 20:06:57.851110 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.850653    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"
Jul 13 20:06:57.851110 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.850655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" event={"ID":"11057863-d56c-4768-80cb-3042cbfe1bd7","Type":"ContainerDied","Data":"78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5"}
Jul 13 20:06:57.851110 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.850813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h" event={"ID":"11057863-d56c-4768-80cb-3042cbfe1bd7","Type":"ContainerDied","Data":"1e9b75515dab277afb8f96a11a97041dd1e55cd35ff47496a5667be1de59c3fb"}
Jul 13 20:06:57.851110 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.850856    2581 scope.go:117] "RemoveContainer" containerID="78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5"
Jul 13 20:06:57.859935 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.859918    2581 scope.go:117] "RemoveContainer" containerID="e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5"
Jul 13 20:06:57.868430 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.868367    2581 scope.go:117] "RemoveContainer" containerID="78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5"
Jul 13 20:06:57.868902 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:06:57.868871    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5\": container with ID starting with 78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5 not found: ID does not exist" containerID="78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5"
Jul 13 20:06:57.868990 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.868915    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5"} err="failed to get container status \"78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5\": rpc error: code = NotFound desc = could not find container \"78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5\": container with ID starting with 78c914672e18589485f519db84d1698fc434e17cf1ac2fbd2b6ed3a9fce3d4d5 not found: ID does not exist"
Jul 13 20:06:57.868990 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.868936    2581 scope.go:117] "RemoveContainer" containerID="e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5"
Jul 13 20:06:57.869210 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:06:57.869193    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5\": container with ID starting with e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5 not found: ID does not exist" containerID="e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5"
Jul 13 20:06:57.869258 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.869216    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5"} err="failed to get container status \"e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5\": rpc error: code = NotFound desc = could not find container \"e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5\": container with ID starting with e583ebd3fdb793edb83861cb865136fbaa279f4794870a71004b8d90c13567d5 not found: ID does not exist"
Jul 13 20:06:57.869524 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.869506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"]
Jul 13 20:06:57.872332 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:57.872311    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-7df7h"]
Jul 13 20:06:59.603161 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:06:59.603115    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" path="/var/lib/kubelet/pods/11057863-d56c-4768-80cb-3042cbfe1bd7/volumes"
Jul 13 20:07:15.679644 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.679563    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"]
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.679966    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="storage-initializer"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.679980    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="storage-initializer"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.679990    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5492c87-cc6f-453c-82d8-9d7e14f17284" containerName="main"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.679995    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5492c87-cc6f-453c-82d8-9d7e14f17284" containerName="main"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680001    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerName="storage-initializer"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680007    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerName="storage-initializer"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680017    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerName="main"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680023    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerName="main"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680030    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main"
Jul 13 20:07:15.680028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680035    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main"
Jul 13 20:07:15.680382 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680086    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="834c278a-adc6-494c-833f-9ed1e843dcb1" containerName="main"
Jul 13 20:07:15.680382 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680093    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="11057863-d56c-4768-80cb-3042cbfe1bd7" containerName="main"
Jul 13 20:07:15.680382 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.680099    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5492c87-cc6f-453c-82d8-9d7e14f17284" containerName="main"
Jul 13 20:07:15.683095 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.683074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.685205 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.685177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 13 20:07:15.685205 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.685193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:07:15.696342 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.696319    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"]
Jul 13 20:07:15.745982 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.745952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:07:15.803041 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.803000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tls-certs\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.803041 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.803045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-home\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.803269 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.803068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tmp-dir\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.803269 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.803175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kserve-provision-location\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.803269 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.803243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7lr6\" (UniqueName: \"kubernetes.io/projected/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kube-api-access-m7lr6\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.803397 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.803273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-dshm\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.803397 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.803321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-model-cache\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.823192 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.823160    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"]
Jul 13 20:07:15.826690 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.826671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:15.828911 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.828892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-9m4n4\""
Jul 13 20:07:15.837184 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.837159    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"]
Jul 13 20:07:15.850612 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.850590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:07:15.904640 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7lr6\" (UniqueName: \"kubernetes.io/projected/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kube-api-access-m7lr6\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.904640 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-dshm\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.904834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-model-cache\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.904834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c210dbd6-9987-4ecb-8781-a73ea313285b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:15.904834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tls-certs\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.904834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:15.904834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:15.905118 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-home\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.905118 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.904969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tmp-dir\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.905118 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.905009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttg96\" (UniqueName: \"kubernetes.io/projected/c210dbd6-9987-4ecb-8781-a73ea313285b-kube-api-access-ttg96\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:15.905118 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.905056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kserve-provision-location\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.905336 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.905118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-model-cache\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.905336 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.905250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kserve-provision-location\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.905336 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.905248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-home\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.905336 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.905303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tmp-dir\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.906934 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.906914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-dshm\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.907319 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.907304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tls-certs\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.913055 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.913032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7lr6\" (UniqueName: \"kubernetes.io/projected/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kube-api-access-m7lr6\") pod \"stop-feature-test-kserve-5f875cbf68-x2lqh\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:15.993543 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:15.993508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:16.006093 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.006062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c210dbd6-9987-4ecb-8781-a73ea313285b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.006211 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.006131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.006211 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.006160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.006334 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.006211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttg96\" (UniqueName: \"kubernetes.io/projected/c210dbd6-9987-4ecb-8781-a73ea313285b-kube-api-access-ttg96\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.006695 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.006663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.006695 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.006686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.009323 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.009298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c210dbd6-9987-4ecb-8781-a73ea313285b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.014739 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.014715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttg96\" (UniqueName: \"kubernetes.io/projected/c210dbd6-9987-4ecb-8781-a73ea313285b-kube-api-access-ttg96\") pod \"stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.135828 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.135773    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"]
Jul 13 20:07:16.136162 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.135944    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:16.137936 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:07:16.137913    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a70840_5e6b_4f6e_b34d_190f123c8cdd.slice/crio-2abc3211ea2dbf4729983612903836b16bb95482db835ab5adc13f7df83c8b88 WatchSource:0}: Error finding container 2abc3211ea2dbf4729983612903836b16bb95482db835ab5adc13f7df83c8b88: Status 404 returned error can't find the container with id 2abc3211ea2dbf4729983612903836b16bb95482db835ab5adc13f7df83c8b88
Jul 13 20:07:16.277095 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.277003    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"]
Jul 13 20:07:16.279863 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:07:16.279838    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc210dbd6_9987_4ecb_8781_a73ea313285b.slice/crio-48bc47036c89c07eccefa26ce84533f8f8c1d4aa53ee9b30b6f8623c27af6167 WatchSource:0}: Error finding container 48bc47036c89c07eccefa26ce84533f8f8c1d4aa53ee9b30b6f8623c27af6167: Status 404 returned error can't find the container with id 48bc47036c89c07eccefa26ce84533f8f8c1d4aa53ee9b30b6f8623c27af6167
Jul 13 20:07:16.945445 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.945409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" event={"ID":"c210dbd6-9987-4ecb-8781-a73ea313285b","Type":"ContainerStarted","Data":"23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764"}
Jul 13 20:07:16.945913 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.945505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" event={"ID":"c210dbd6-9987-4ecb-8781-a73ea313285b","Type":"ContainerStarted","Data":"48bc47036c89c07eccefa26ce84533f8f8c1d4aa53ee9b30b6f8623c27af6167"}
Jul 13 20:07:16.947064 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.947037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" event={"ID":"e4a70840-5e6b-4f6e-b34d-190f123c8cdd","Type":"ContainerStarted","Data":"bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610"}
Jul 13 20:07:16.947166 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:16.947083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" event={"ID":"e4a70840-5e6b-4f6e-b34d-190f123c8cdd","Type":"ContainerStarted","Data":"2abc3211ea2dbf4729983612903836b16bb95482db835ab5adc13f7df83c8b88"}
Jul 13 20:07:17.953503 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:17.953463    2581 generic.go:356] "Generic (PLEG): container finished" podID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerID="23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764" exitCode=0
Jul 13 20:07:17.954237 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:17.953549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" event={"ID":"c210dbd6-9987-4ecb-8781-a73ea313285b","Type":"ContainerDied","Data":"23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764"}
Jul 13 20:07:18.963484 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:18.963452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" event={"ID":"c210dbd6-9987-4ecb-8781-a73ea313285b","Type":"ContainerStarted","Data":"4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19"}
Jul 13 20:07:18.963934 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:18.963562    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:18.982859 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:18.982818    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" podStartSLOduration=3.264262851 podStartE2EDuration="3.982804001s" totalImagesPullingTime="0s" totalInitContainerRuntime="718.54115ms" isStatefulPod=false podCreationTimestamp="2026-07-13 20:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:07:18.98004247 +0000 UTC m=+4675.880267861" watchObservedRunningTime="2026-07-13 20:07:18.982804001 +0000 UTC m=+4675.883029384"
Jul 13 20:07:19.969340 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:19.969306    2581 generic.go:356] "Generic (PLEG): container finished" podID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerID="bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610" exitCode=0
Jul 13 20:07:19.969810 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:19.969378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" event={"ID":"e4a70840-5e6b-4f6e-b34d-190f123c8cdd","Type":"ContainerDied","Data":"bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610"}
Jul 13 20:07:20.975909 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:20.975868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" event={"ID":"e4a70840-5e6b-4f6e-b34d-190f123c8cdd","Type":"ContainerStarted","Data":"1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf"}
Jul 13 20:07:20.994024 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:20.993989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:21.000327 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:21.000271    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podStartSLOduration=3.069329245 podStartE2EDuration="6.000252432s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.930923187s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:07:20.999128389 +0000 UTC m=+4677.899353803" watchObservedRunningTime="2026-07-13 20:07:21.000252432 +0000 UTC m=+4677.900477825"
Jul 13 20:07:25.994527 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:25.994486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:07:25.996229 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:25.996195    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:07:35.994362 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:35.994321    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:07:45.994328 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:45.994281    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:07:49.971154 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:49.971124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:07:55.994965 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:07:55.994912    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:08:05.093645 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:05.093611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:08:05.194475 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:05.194405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:08:05.994365 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:05.994325    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:08:15.994894 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:15.994829    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:08:20.391707 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:20.391663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:08:25.994502 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:25.994452    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:08:35.993966 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:35.993914    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:08:45.995020 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:45.994926    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:08:52.317680 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:52.317644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:08:55.994310 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:55.994258    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:08:56.827927 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:08:56.827885    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:08:57.511743 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:57.511710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:08:58.292223 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:08:58.292191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:09:05.995009 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:09:05.994955    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:09:16.004735 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:09:16.004706    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:09:16.012645 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:09:16.012617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:09:23.979852 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:09:23.979825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:09:23.993253 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:09:23.993235    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:09:32.062952 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:09:32.062896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:09:32.129001 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:09:32.128969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:10:17.605546 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.605469    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"]
Jul 13 20:10:17.611259 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.610588    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"]
Jul 13 20:10:17.617716 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.617689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:10:17.618603 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:10:17.618567    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:10:17.618603 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:10:17.618603 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:10:17.618603 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19" cmd=["/bin/sleep","15"]
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.618625    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" message="" err=<
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:10:17.618642    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerName="main" containerID="cri-o://4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19"
Jul 13 20:10:17.618929 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.618699    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerName="main" containerID="cri-o://4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19" gracePeriod=60
Jul 13 20:10:17.717808 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.717744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:10:17.871001 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.870979    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:10:17.924194 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.924158    2581 generic.go:356] "Generic (PLEG): container finished" podID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerID="4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19" exitCode=0
Jul 13 20:10:17.924377 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.924207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"
Jul 13 20:10:17.924377 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.924245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" event={"ID":"c210dbd6-9987-4ecb-8781-a73ea313285b","Type":"ContainerDied","Data":"4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19"}
Jul 13 20:10:17.924377 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.924300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9" event={"ID":"c210dbd6-9987-4ecb-8781-a73ea313285b","Type":"ContainerDied","Data":"48bc47036c89c07eccefa26ce84533f8f8c1d4aa53ee9b30b6f8623c27af6167"}
Jul 13 20:10:17.924377 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.924329    2581 scope.go:117] "RemoveContainer" containerID="4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19"
Jul 13 20:10:17.934815 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.934775    2581 scope.go:117] "RemoveContainer" containerID="23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764"
Jul 13 20:10:17.943783 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.943764    2581 scope.go:117] "RemoveContainer" containerID="4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19"
Jul 13 20:10:17.944091 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:10:17.944068    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19\": container with ID starting with 4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19 not found: ID does not exist" containerID="4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19"
Jul 13 20:10:17.944176 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.944101    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19"} err="failed to get container status \"4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19\": rpc error: code = NotFound desc = could not find container \"4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19\": container with ID starting with 4bea5dc766662efd28c2d01e701d116fa5304def468a31135af54add0ac0dd19 not found: ID does not exist"
Jul 13 20:10:17.944176 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.944124    2581 scope.go:117] "RemoveContainer" containerID="23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764"
Jul 13 20:10:17.944381 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:10:17.944364    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764\": container with ID starting with 23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764 not found: ID does not exist" containerID="23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764"
Jul 13 20:10:17.944428 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.944387    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764"} err="failed to get container status \"23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764\": rpc error: code = NotFound desc = could not find container \"23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764\": container with ID starting with 23d71e1d5c2bed337093fac2060fa72789d2ff1ab231a7bd0fec9d1921c88764 not found: ID does not exist"
Jul 13 20:10:17.951733 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.951710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-tokenizer-uds\") pod \"c210dbd6-9987-4ecb-8781-a73ea313285b\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") "
Jul 13 20:10:17.951828 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.951750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttg96\" (UniqueName: \"kubernetes.io/projected/c210dbd6-9987-4ecb-8781-a73ea313285b-kube-api-access-ttg96\") pod \"c210dbd6-9987-4ecb-8781-a73ea313285b\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") "
Jul 13 20:10:17.951828 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.951781    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c210dbd6-9987-4ecb-8781-a73ea313285b-tls-certs\") pod \"c210dbd6-9987-4ecb-8781-a73ea313285b\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") "
Jul 13 20:10:17.951939 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.951917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-kserve-provision-location\") pod \"c210dbd6-9987-4ecb-8781-a73ea313285b\" (UID: \"c210dbd6-9987-4ecb-8781-a73ea313285b\") "
Jul 13 20:10:17.952024 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.951998    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c210dbd6-9987-4ecb-8781-a73ea313285b" (UID: "c210dbd6-9987-4ecb-8781-a73ea313285b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:10:17.952192 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.952174    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:17.952689 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.952666    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c210dbd6-9987-4ecb-8781-a73ea313285b" (UID: "c210dbd6-9987-4ecb-8781-a73ea313285b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:10:17.954461 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.954441    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c210dbd6-9987-4ecb-8781-a73ea313285b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c210dbd6-9987-4ecb-8781-a73ea313285b" (UID: "c210dbd6-9987-4ecb-8781-a73ea313285b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:10:17.954531 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:17.954456    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c210dbd6-9987-4ecb-8781-a73ea313285b-kube-api-access-ttg96" (OuterVolumeSpecName: "kube-api-access-ttg96") pod "c210dbd6-9987-4ecb-8781-a73ea313285b" (UID: "c210dbd6-9987-4ecb-8781-a73ea313285b"). InnerVolumeSpecName "kube-api-access-ttg96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:10:18.052805 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:18.052767    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttg96\" (UniqueName: \"kubernetes.io/projected/c210dbd6-9987-4ecb-8781-a73ea313285b-kube-api-access-ttg96\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:18.052970 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:18.052820    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c210dbd6-9987-4ecb-8781-a73ea313285b-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:18.052970 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:18.052830    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c210dbd6-9987-4ecb-8781-a73ea313285b-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:18.260095 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:18.260063    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"]
Jul 13 20:10:18.264720 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:18.264689    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76cf8bfb8f-qgsf9"]
Jul 13 20:10:19.604003 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:19.603968    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" path="/var/lib/kubelet/pods/c210dbd6-9987-4ecb-8781-a73ea313285b/volumes"
Jul 13 20:10:32.640429 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:32.640366    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" containerID="cri-o://1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf" gracePeriod=45
Jul 13 20:10:33.005731 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.005691    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 13 20:10:33.398742 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.398720    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:10:33.487751 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.487717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-home\") pod \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") "
Jul 13 20:10:33.487751 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.487749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-dshm\") pod \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") "
Jul 13 20:10:33.487967 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.487770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kserve-provision-location\") pod \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") "
Jul 13 20:10:33.487967 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.487809    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tmp-dir\") pod \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") "
Jul 13 20:10:33.487967 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.487864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7lr6\" (UniqueName: \"kubernetes.io/projected/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kube-api-access-m7lr6\") pod \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") "
Jul 13 20:10:33.487967 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.487895    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tls-certs\") pod \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") "
Jul 13 20:10:33.487967 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.487957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-model-cache\") pod \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\" (UID: \"e4a70840-5e6b-4f6e-b34d-190f123c8cdd\") "
Jul 13 20:10:33.488436 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.488395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4a70840-5e6b-4f6e-b34d-190f123c8cdd" (UID: "e4a70840-5e6b-4f6e-b34d-190f123c8cdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:10:33.488566 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.488529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-home" (OuterVolumeSpecName: "home") pod "e4a70840-5e6b-4f6e-b34d-190f123c8cdd" (UID: "e4a70840-5e6b-4f6e-b34d-190f123c8cdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:10:33.490236 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.490204    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4a70840-5e6b-4f6e-b34d-190f123c8cdd" (UID: "e4a70840-5e6b-4f6e-b34d-190f123c8cdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:10:33.490484 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.490440    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kube-api-access-m7lr6" (OuterVolumeSpecName: "kube-api-access-m7lr6") pod "e4a70840-5e6b-4f6e-b34d-190f123c8cdd" (UID: "e4a70840-5e6b-4f6e-b34d-190f123c8cdd"). InnerVolumeSpecName "kube-api-access-m7lr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:10:33.490595 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.490510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-dshm" (OuterVolumeSpecName: "dshm") pod "e4a70840-5e6b-4f6e-b34d-190f123c8cdd" (UID: "e4a70840-5e6b-4f6e-b34d-190f123c8cdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:10:33.500990 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.500964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4a70840-5e6b-4f6e-b34d-190f123c8cdd" (UID: "e4a70840-5e6b-4f6e-b34d-190f123c8cdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:10:33.552754 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.552671    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4a70840-5e6b-4f6e-b34d-190f123c8cdd" (UID: "e4a70840-5e6b-4f6e-b34d-190f123c8cdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:10:33.589225 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.589185    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7lr6\" (UniqueName: \"kubernetes.io/projected/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kube-api-access-m7lr6\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:33.589225 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.589218    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:33.589225 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.589232    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:33.589500 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.589243    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:33.589500 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.589253    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:33.589500 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.589265    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:33.589500 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:33.589275    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4a70840-5e6b-4f6e-b34d-190f123c8cdd-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:10:34.001133 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.001089    2581 generic.go:356] "Generic (PLEG): container finished" podID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerID="1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf" exitCode=0
Jul 13 20:10:34.001553 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.001137    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"
Jul 13 20:10:34.001553 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.001159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" event={"ID":"e4a70840-5e6b-4f6e-b34d-190f123c8cdd","Type":"ContainerDied","Data":"1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf"}
Jul 13 20:10:34.001553 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.001211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh" event={"ID":"e4a70840-5e6b-4f6e-b34d-190f123c8cdd","Type":"ContainerDied","Data":"2abc3211ea2dbf4729983612903836b16bb95482db835ab5adc13f7df83c8b88"}
Jul 13 20:10:34.001553 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.001235    2581 scope.go:117] "RemoveContainer" containerID="1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf"
Jul 13 20:10:34.014988 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.014963    2581 scope.go:117] "RemoveContainer" containerID="bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610"
Jul 13 20:10:34.024629 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.024605    2581 scope.go:117] "RemoveContainer" containerID="1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf"
Jul 13 20:10:34.025029 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:10:34.024973    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf\": container with ID starting with 1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf not found: ID does not exist" containerID="1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf"
Jul 13 20:10:34.025201 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.025051    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf"} err="failed to get container status \"1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf\": rpc error: code = NotFound desc = could not find container \"1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf\": container with ID starting with 1b541b65861d4e4e382acd7dcbe190a9fcff084e09c64318fd85dfeb1b88accf not found: ID does not exist"
Jul 13 20:10:34.025201 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.025080    2581 scope.go:117] "RemoveContainer" containerID="bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610"
Jul 13 20:10:34.025415 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.025280    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"]
Jul 13 20:10:34.025415 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:10:34.025390    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610\": container with ID starting with bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610 not found: ID does not exist" containerID="bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610"
Jul 13 20:10:34.025525 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.025424    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610"} err="failed to get container status \"bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610\": rpc error: code = NotFound desc = could not find container \"bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610\": container with ID starting with bc0b3d77094c8f85fbee55df72462c0326dcee9296459ea497989db5b4549610 not found: ID does not exist"
Jul 13 20:10:34.027243 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:34.027223    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5f875cbf68-x2lqh"]
Jul 13 20:10:35.603110 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:35.603073    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" path="/var/lib/kubelet/pods/e4a70840-5e6b-4f6e-b34d-190f123c8cdd/volumes"
Jul 13 20:10:41.396579 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396547    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"]
Jul 13 20:10:41.396959 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396911    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerName="storage-initializer"
Jul 13 20:10:41.396959 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396924    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerName="storage-initializer"
Jul 13 20:10:41.396959 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396941    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="storage-initializer"
Jul 13 20:10:41.396959 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396946    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="storage-initializer"
Jul 13 20:10:41.396959 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396954    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerName="main"
Jul 13 20:10:41.396959 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396960    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerName="main"
Jul 13 20:10:41.397154 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396966    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main"
Jul 13 20:10:41.397154 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.396970    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main"
Jul 13 20:10:41.397154 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.397020    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c210dbd6-9987-4ecb-8781-a73ea313285b" containerName="main"
Jul 13 20:10:41.397154 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.397029    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a70840-5e6b-4f6e-b34d-190f123c8cdd" containerName="main"
Jul 13 20:10:41.401543 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.401524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.404101 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.404071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:10:41.404212 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.404112    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 13 20:10:41.416718 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.416693    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"]
Jul 13 20:10:41.431968 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.431942    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:10:41.454803 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.454759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-dshm\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.454909 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.454830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tls-certs\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.454909 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.454874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kserve-provision-location\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.454988 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.454920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tmp-dir\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.454988 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.454959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-model-cache\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.455050 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.455005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-home\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.455087 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.455054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlt2p\" (UniqueName: \"kubernetes.io/projected/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kube-api-access-zlt2p\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.555506 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.555469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-dshm\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.555728 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.555523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tls-certs\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.555728 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.555554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kserve-provision-location\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.555728 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.555571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tmp-dir\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.555728 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.555595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-model-cache\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.555728 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.555628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-home\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.555728 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.555667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlt2p\" (UniqueName: \"kubernetes.io/projected/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kube-api-access-zlt2p\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.556102 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.556063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kserve-provision-location\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.556102 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.556075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tmp-dir\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.556200 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.556131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-model-cache\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.556200 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.556161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-home\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.558043 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.558026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-dshm\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.558304 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.558287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tls-certs\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.566259 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.566235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlt2p\" (UniqueName: \"kubernetes.io/projected/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kube-api-access-zlt2p\") pod \"prestop-hook-test-kserve-65cf7fd568-kgw6w\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.711316 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.711282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:41.835461 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.835429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"]
Jul 13 20:10:41.839908 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.839882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:41.842669 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.842520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-hhrv7\""
Jul 13 20:10:41.854306 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.854279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"]
Jul 13 20:10:41.868424 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.868395    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"]
Jul 13 20:10:41.869062 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:10:41.869039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9943c2_679f_417e_b5bb_fe88aaf8a2a9.slice/crio-9457f192bb23aa91fbe06742cedd0506b01e7452172449c6e54cde02abc3823a WatchSource:0}: Error finding container 9457f192bb23aa91fbe06742cedd0506b01e7452172449c6e54cde02abc3823a: Status 404 returned error can't find the container with id 9457f192bb23aa91fbe06742cedd0506b01e7452172449c6e54cde02abc3823a
Jul 13 20:10:41.888445 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.888426    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:10:41.959290 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.959251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:41.959546 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.959523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99cafc68-f538-44f0-98a9-1b317239a7a5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:41.959711 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.959691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:41.959982 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:41.959875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnsb\" (UniqueName: \"kubernetes.io/projected/99cafc68-f538-44f0-98a9-1b317239a7a5-kube-api-access-xwnsb\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.042541 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.042448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" event={"ID":"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9","Type":"ContainerStarted","Data":"c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb"}
Jul 13 20:10:42.042541 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.042497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" event={"ID":"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9","Type":"ContainerStarted","Data":"9457f192bb23aa91fbe06742cedd0506b01e7452172449c6e54cde02abc3823a"}
Jul 13 20:10:42.061041 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.060999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnsb\" (UniqueName: \"kubernetes.io/projected/99cafc68-f538-44f0-98a9-1b317239a7a5-kube-api-access-xwnsb\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.061228 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.061083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.061228 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.061105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99cafc68-f538-44f0-98a9-1b317239a7a5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.061228 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.061128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.061458 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.061441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.061524 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.061485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.063636 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.063613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99cafc68-f538-44f0-98a9-1b317239a7a5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.072045 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.072018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnsb\" (UniqueName: \"kubernetes.io/projected/99cafc68-f538-44f0-98a9-1b317239a7a5-kube-api-access-xwnsb\") pod \"prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.150397 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.150348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:42.298972 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:42.297429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"]
Jul 13 20:10:43.050973 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:43.050930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" event={"ID":"99cafc68-f538-44f0-98a9-1b317239a7a5","Type":"ContainerStarted","Data":"a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8"}
Jul 13 20:10:43.051516 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:43.050985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" event={"ID":"99cafc68-f538-44f0-98a9-1b317239a7a5","Type":"ContainerStarted","Data":"077ad7de23795a399d4aee28ee6b1b4564b88b5a362aba60b899fea60ef75998"}
Jul 13 20:10:44.066643 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:44.066599    2581 generic.go:356] "Generic (PLEG): container finished" podID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerID="a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8" exitCode=0
Jul 13 20:10:44.067176 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:44.066682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" event={"ID":"99cafc68-f538-44f0-98a9-1b317239a7a5","Type":"ContainerDied","Data":"a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8"}
Jul 13 20:10:45.073959 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:45.073928    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerID="c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb" exitCode=0
Jul 13 20:10:45.074332 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:45.074001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" event={"ID":"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9","Type":"ContainerDied","Data":"c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb"}
Jul 13 20:10:45.076039 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:45.076005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" event={"ID":"99cafc68-f538-44f0-98a9-1b317239a7a5","Type":"ContainerStarted","Data":"a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9"}
Jul 13 20:10:45.076156 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:45.076137    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:10:45.127302 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:45.127249    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" podStartSLOduration=3.368456952 podStartE2EDuration="4.127228647s" totalImagesPullingTime="0s" totalInitContainerRuntime="758.771695ms" isStatefulPod=false podCreationTimestamp="2026-07-13 20:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:10:45.123552452 +0000 UTC m=+4882.023777845" watchObservedRunningTime="2026-07-13 20:10:45.127228647 +0000 UTC m=+4882.027454040"
Jul 13 20:10:46.083727 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:46.083696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" event={"ID":"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9","Type":"ContainerStarted","Data":"e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2"}
Jul 13 20:10:46.114499 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:46.114448    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podStartSLOduration=2.039243096 podStartE2EDuration="5.114430848s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.075187752s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:10:46.112296841 +0000 UTC m=+4883.012522234" watchObservedRunningTime="2026-07-13 20:10:46.114430848 +0000 UTC m=+4883.014656239"
Jul 13 20:10:46.712164 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:46.712118    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:51.711683 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:51.711644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:10:51.713356 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:51.713327    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:10:58.671882 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:10:58.671839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:11:00.513108 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:11:00.513061    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:11:01.160900 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:01.160871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:11:01.712725 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:01.712684    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:11:11.712734 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:11.712690    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:11:16.085557 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:16.085522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:11:21.712605 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:21.712561    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:11:25.627834 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.627773    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"]
Jul 13 20:11:25.632739 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.632712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.635509 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.635477    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 20:11:25.635637 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.635547    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 13 20:11:25.649694 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.649661    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"]
Jul 13 20:11:25.666470 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.666425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:11:25.675747 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.675702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.675747 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.675747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4t98\" (UniqueName: \"kubernetes.io/projected/56a14432-596d-48e6-a8c5-ad5434d07ca2-kube-api-access-g4t98\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.676005 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.675823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-dshm\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.676005 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.675842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a14432-596d-48e6-a8c5-ad5434d07ca2-tls-certs\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.676005 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.675929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a14432-596d-48e6-a8c5-ad5434d07ca2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.676005 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.675977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-home\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.676231 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.676021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-model-cache\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.676231 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.676050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-tmp-dir\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777316 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4t98\" (UniqueName: \"kubernetes.io/projected/56a14432-596d-48e6-a8c5-ad5434d07ca2-kube-api-access-g4t98\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777316 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-dshm\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777607 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a14432-596d-48e6-a8c5-ad5434d07ca2-tls-certs\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777607 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a14432-596d-48e6-a8c5-ad5434d07ca2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777607 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-home\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777607 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-model-cache\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777607 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-tmp-dir\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777607 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777994 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-model-cache\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777994 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777994 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-tmp-dir\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.777994 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.777933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-home\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.778693 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.778669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a14432-596d-48e6-a8c5-ad5434d07ca2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.779830 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.779808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-dshm\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.780284 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.780265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a14432-596d-48e6-a8c5-ad5434d07ca2-tls-certs\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.784946 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.784922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4t98\" (UniqueName: \"kubernetes.io/projected/56a14432-596d-48e6-a8c5-ad5434d07ca2-kube-api-access-g4t98\") pod \"lora-single-adapter-test-kserve-5849bd8489-gtjrb\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:25.944409 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:25.944307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:26.098681 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:26.098590    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"]
Jul 13 20:11:26.100967 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:11:26.100941    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a14432_596d_48e6_a8c5_ad5434d07ca2.slice/crio-f68c90590f0e64ca332124c2618aab3d4c6a968e6c692405f0dcd5fd67ff3ea8 WatchSource:0}: Error finding container f68c90590f0e64ca332124c2618aab3d4c6a968e6c692405f0dcd5fd67ff3ea8: Status 404 returned error can't find the container with id f68c90590f0e64ca332124c2618aab3d4c6a968e6c692405f0dcd5fd67ff3ea8
Jul 13 20:11:26.288807 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:26.288756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" event={"ID":"56a14432-596d-48e6-a8c5-ad5434d07ca2","Type":"ContainerStarted","Data":"3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376"}
Jul 13 20:11:26.289012 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:26.288830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" event={"ID":"56a14432-596d-48e6-a8c5-ad5434d07ca2","Type":"ContainerStarted","Data":"f68c90590f0e64ca332124c2618aab3d4c6a968e6c692405f0dcd5fd67ff3ea8"}
Jul 13 20:11:30.313138 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:30.313097    2581 generic.go:356] "Generic (PLEG): container finished" podID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerID="3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376" exitCode=0
Jul 13 20:11:30.313631 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:30.313171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" event={"ID":"56a14432-596d-48e6-a8c5-ad5434d07ca2","Type":"ContainerDied","Data":"3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376"}
Jul 13 20:11:31.320942 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:31.320897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" event={"ID":"56a14432-596d-48e6-a8c5-ad5434d07ca2","Type":"ContainerStarted","Data":"f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb"}
Jul 13 20:11:31.346934 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:31.346876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podStartSLOduration=3.01597807 podStartE2EDuration="6.346858165s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.330880095s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:11:31.344856046 +0000 UTC m=+4928.245081438" watchObservedRunningTime="2026-07-13 20:11:31.346858165 +0000 UTC m=+4928.247083558"
Jul 13 20:11:31.712326 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:31.712290    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:11:31.944638 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:31.944602    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:35.945472 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:35.945430    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:11:35.947377 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:35.947346    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:11:36.978125 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:36.978097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:11:41.711938 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:41.711899    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:11:45.683856 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:45.683732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:11:45.945468 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:45.945378    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:11:51.712216 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:51.712168    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:11:55.945154 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:11:55.945103    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:12:01.712344 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:01.712297    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:12:05.945621 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:05.945563    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:12:11.712039 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:11.711994    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:12:15.945007 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:15.944960    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:12:21.712402 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:21.712352    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:12:25.945020 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:25.944973    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:12:31.712584 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:31.712541    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:12:35.945150 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:35.945106    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:12:41.722688 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:41.722655    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:12:41.731428 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:41.731397    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:12:44.394854 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:44.394782    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"]
Jul 13 20:12:45.944998 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:45.944960    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 13 20:12:55.954991 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:55.954961    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:12:55.962671 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:55.962642    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:12:59.435329 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:59.435285    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" containerID="cri-o://e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2" gracePeriod=45
Jul 13 20:12:59.723562 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:12:59.723505    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 13 20:13:00.417665 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.417640    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:13:00.479334 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.479304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-dshm\") pod \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") "
Jul 13 20:13:00.479845 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.479349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-home\") pod \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") "
Jul 13 20:13:00.479845 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.479405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tls-certs\") pod \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") "
Jul 13 20:13:00.479845 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.479454    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tmp-dir\") pod \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") "
Jul 13 20:13:00.479845 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.479489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kserve-provision-location\") pod \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") "
Jul 13 20:13:00.479845 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.479531    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-model-cache\") pod \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") "
Jul 13 20:13:00.479845 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.479569    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlt2p\" (UniqueName: \"kubernetes.io/projected/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kube-api-access-zlt2p\") pod \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\" (UID: \"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9\") "
Jul 13 20:13:00.480190 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.480008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" (UID: "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:13:00.480190 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.480034    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-home" (OuterVolumeSpecName: "home") pod "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" (UID: "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:13:00.482973 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.482907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-dshm" (OuterVolumeSpecName: "dshm") pod "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" (UID: "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:13:00.483295 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.483268    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kube-api-access-zlt2p" (OuterVolumeSpecName: "kube-api-access-zlt2p") pod "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" (UID: "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9"). InnerVolumeSpecName "kube-api-access-zlt2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:13:00.483572 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.483542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" (UID: "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:13:00.497699 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.497631    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" (UID: "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:13:00.558924 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.558871    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" (UID: "fb9943c2-679f-417e-b5bb-fe88aaf8a2a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:13:00.580506 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.580473    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:00.580506 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.580501    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlt2p\" (UniqueName: \"kubernetes.io/projected/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kube-api-access-zlt2p\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:00.580506 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.580511    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:00.580745 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.580519    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:00.580745 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.580531    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:00.580745 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.580544    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:00.580745 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.580552    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:00.852630 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.852539    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerID="e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2" exitCode=0
Jul 13 20:13:00.852938 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.852588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" event={"ID":"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9","Type":"ContainerDied","Data":"e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2"}
Jul 13 20:13:00.853085 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.852986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w" event={"ID":"fb9943c2-679f-417e-b5bb-fe88aaf8a2a9","Type":"ContainerDied","Data":"9457f192bb23aa91fbe06742cedd0506b01e7452172449c6e54cde02abc3823a"}
Jul 13 20:13:00.853085 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.852622    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"
Jul 13 20:13:00.853085 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.853029    2581 scope.go:117] "RemoveContainer" containerID="e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2"
Jul 13 20:13:00.865065 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.865045    2581 scope.go:117] "RemoveContainer" containerID="c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb"
Jul 13 20:13:00.876228 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.876205    2581 scope.go:117] "RemoveContainer" containerID="e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2"
Jul 13 20:13:00.876558 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:13:00.876536    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2\": container with ID starting with e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2 not found: ID does not exist" containerID="e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2"
Jul 13 20:13:00.876683 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.876564    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2"} err="failed to get container status \"e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2\": rpc error: code = NotFound desc = could not find container \"e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2\": container with ID starting with e7400ba07f1f62646f49c4231852816e44615667c2e65f96e25127a4232094a2 not found: ID does not exist"
Jul 13 20:13:00.876683 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.876582    2581 scope.go:117] "RemoveContainer" containerID="c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb"
Jul 13 20:13:00.876915 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:13:00.876856    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb\": container with ID starting with c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb not found: ID does not exist" containerID="c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb"
Jul 13 20:13:00.876915 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.876874    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb"} err="failed to get container status \"c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb\": rpc error: code = NotFound desc = could not find container \"c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb\": container with ID starting with c0ae1c6b516dbf9a312fccfe5dad83beee4b2e8853d20b2920df8f5dd6c3cedb not found: ID does not exist"
Jul 13 20:13:00.889458 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.889432    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"]
Jul 13 20:13:00.894068 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:00.894042    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-kgw6w"]
Jul 13 20:13:01.603500 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:01.603463    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" path="/var/lib/kubelet/pods/fb9943c2-679f-417e-b5bb-fe88aaf8a2a9/volumes"
Jul 13 20:13:04.162230 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:13:04.162175    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:13:04.295942 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:04.295900    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:13:04.875273 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:04.875237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:13:07.983652 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:07.983609    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:13:58.911366 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:58.911328    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"]
Jul 13 20:13:59.005025 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.004986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:13:59.240122 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.240082    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"]
Jul 13 20:13:59.249604 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:13:59.249561    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:13:59.249604 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:13:59.249604 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:13:59.249604 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9" cmd=["/bin/sleep","15"]
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.249612    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" message="" err=<
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:13:59.249625    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerName="main" containerID="cri-o://a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9"
Jul 13 20:13:59.249871 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.249672    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerName="main" containerID="cri-o://a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9" gracePeriod=60
Jul 13 20:13:59.500848 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.500769    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:13:59.614642 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.614613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwnsb\" (UniqueName: \"kubernetes.io/projected/99cafc68-f538-44f0-98a9-1b317239a7a5-kube-api-access-xwnsb\") pod \"99cafc68-f538-44f0-98a9-1b317239a7a5\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") "
Jul 13 20:13:59.614843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.614654    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-tokenizer-uds\") pod \"99cafc68-f538-44f0-98a9-1b317239a7a5\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") "
Jul 13 20:13:59.614843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.614724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99cafc68-f538-44f0-98a9-1b317239a7a5-tls-certs\") pod \"99cafc68-f538-44f0-98a9-1b317239a7a5\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") "
Jul 13 20:13:59.614843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.614827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-kserve-provision-location\") pod \"99cafc68-f538-44f0-98a9-1b317239a7a5\" (UID: \"99cafc68-f538-44f0-98a9-1b317239a7a5\") "
Jul 13 20:13:59.615034 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.615005    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "99cafc68-f538-44f0-98a9-1b317239a7a5" (UID: "99cafc68-f538-44f0-98a9-1b317239a7a5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:13:59.615090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.615068    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:59.615627 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.615595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99cafc68-f538-44f0-98a9-1b317239a7a5" (UID: "99cafc68-f538-44f0-98a9-1b317239a7a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:13:59.617003 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.616971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cafc68-f538-44f0-98a9-1b317239a7a5-kube-api-access-xwnsb" (OuterVolumeSpecName: "kube-api-access-xwnsb") pod "99cafc68-f538-44f0-98a9-1b317239a7a5" (UID: "99cafc68-f538-44f0-98a9-1b317239a7a5"). InnerVolumeSpecName "kube-api-access-xwnsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:13:59.617003 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.616970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99cafc68-f538-44f0-98a9-1b317239a7a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99cafc68-f538-44f0-98a9-1b317239a7a5" (UID: "99cafc68-f538-44f0-98a9-1b317239a7a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:13:59.716395 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.716361    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99cafc68-f538-44f0-98a9-1b317239a7a5-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:59.716395 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.716396    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwnsb\" (UniqueName: \"kubernetes.io/projected/99cafc68-f538-44f0-98a9-1b317239a7a5-kube-api-access-xwnsb\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:13:59.716395 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:13:59.716414    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99cafc68-f538-44f0-98a9-1b317239a7a5-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:00.179679 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.179648    2581 generic.go:356] "Generic (PLEG): container finished" podID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerID="a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9" exitCode=0
Jul 13 20:14:00.180112 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.179712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" event={"ID":"99cafc68-f538-44f0-98a9-1b317239a7a5","Type":"ContainerDied","Data":"a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9"}
Jul 13 20:14:00.180112 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.179717    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"
Jul 13 20:14:00.180112 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.179752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2" event={"ID":"99cafc68-f538-44f0-98a9-1b317239a7a5","Type":"ContainerDied","Data":"077ad7de23795a399d4aee28ee6b1b4564b88b5a362aba60b899fea60ef75998"}
Jul 13 20:14:00.180112 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.179771    2581 scope.go:117] "RemoveContainer" containerID="a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9"
Jul 13 20:14:00.181988 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.181958    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:00.189698 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.189684    2581 scope.go:117] "RemoveContainer" containerID="a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8"
Jul 13 20:14:00.197537 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.197509    2581 scope.go:117] "RemoveContainer" containerID="a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9"
Jul 13 20:14:00.197826 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:14:00.197800    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9\": container with ID starting with a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9 not found: ID does not exist" containerID="a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9"
Jul 13 20:14:00.197926 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.197833    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9"} err="failed to get container status \"a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9\": rpc error: code = NotFound desc = could not find container \"a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9\": container with ID starting with a564ffc09afe049f7ea1dacfb0ed1655266a588c6b77ce1c934d6b97f136b7c9 not found: ID does not exist"
Jul 13 20:14:00.197926 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.197849    2581 scope.go:117] "RemoveContainer" containerID="a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8"
Jul 13 20:14:00.198100 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:14:00.198080    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8\": container with ID starting with a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8 not found: ID does not exist" containerID="a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8"
Jul 13 20:14:00.198156 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.198109    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8"} err="failed to get container status \"a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8\": rpc error: code = NotFound desc = could not find container \"a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8\": container with ID starting with a0fc09d72bbe84f04b267d37a5d2c8d131f6208dc74be5924034cda7746b7db8 not found: ID does not exist"
Jul 13 20:14:00.247590 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.247559    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"]
Jul 13 20:14:00.260254 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.260225    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-86bc684b85-qqhf2"]
Jul 13 20:14:00.386391 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:00.386359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:01.604145 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:01.604107    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" path="/var/lib/kubelet/pods/99cafc68-f538-44f0-98a9-1b317239a7a5/volumes"
Jul 13 20:14:14.202732 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.202670    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main" containerID="cri-o://f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb" gracePeriod=45
Jul 13 20:14:14.846294 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.846270    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:14:14.951719 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-home\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.951719 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-tmp-dir\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.951719 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-dshm\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.952030 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-kserve-provision-location\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.952030 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a14432-596d-48e6-a8c5-ad5434d07ca2-cabundle-cert\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.952030 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a14432-596d-48e6-a8c5-ad5434d07ca2-tls-certs\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.952030 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4t98\" (UniqueName: \"kubernetes.io/projected/56a14432-596d-48e6-a8c5-ad5434d07ca2-kube-api-access-g4t98\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.952030 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.951919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-model-cache\") pod \"56a14432-596d-48e6-a8c5-ad5434d07ca2\" (UID: \"56a14432-596d-48e6-a8c5-ad5434d07ca2\") "
Jul 13 20:14:14.952287 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.952111    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-home" (OuterVolumeSpecName: "home") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:14:14.952287 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.952218    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:14.952287 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.952208    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a14432-596d-48e6-a8c5-ad5434d07ca2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:14:14.952443 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.952352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-model-cache" (OuterVolumeSpecName: "model-cache") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:14:14.954157 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.954107    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-dshm" (OuterVolumeSpecName: "dshm") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:14:14.954463 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.954432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a14432-596d-48e6-a8c5-ad5434d07ca2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:14:14.954554 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.954534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a14432-596d-48e6-a8c5-ad5434d07ca2-kube-api-access-g4t98" (OuterVolumeSpecName: "kube-api-access-g4t98") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "kube-api-access-g4t98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:14:14.966519 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:14.966497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:14:15.008494 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.008452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56a14432-596d-48e6-a8c5-ad5434d07ca2" (UID: "56a14432-596d-48e6-a8c5-ad5434d07ca2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:14:15.053364 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.053335    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:15.053364 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.053360    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a14432-596d-48e6-a8c5-ad5434d07ca2-cabundle-cert\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:15.053559 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.053383    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a14432-596d-48e6-a8c5-ad5434d07ca2-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:15.053559 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.053395    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4t98\" (UniqueName: \"kubernetes.io/projected/56a14432-596d-48e6-a8c5-ad5434d07ca2-kube-api-access-g4t98\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:15.053559 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.053406    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:15.053559 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.053418    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:15.053559 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.053428    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a14432-596d-48e6-a8c5-ad5434d07ca2-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:14:15.257358 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.257327    2581 generic.go:356] "Generic (PLEG): container finished" podID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerID="f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb" exitCode=0
Jul 13 20:14:15.257859 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.257398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"
Jul 13 20:14:15.257859 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.257426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" event={"ID":"56a14432-596d-48e6-a8c5-ad5434d07ca2","Type":"ContainerDied","Data":"f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb"}
Jul 13 20:14:15.257859 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.257482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb" event={"ID":"56a14432-596d-48e6-a8c5-ad5434d07ca2","Type":"ContainerDied","Data":"f68c90590f0e64ca332124c2618aab3d4c6a968e6c692405f0dcd5fd67ff3ea8"}
Jul 13 20:14:15.257859 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.257506    2581 scope.go:117] "RemoveContainer" containerID="f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb"
Jul 13 20:14:15.266832 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.266815    2581 scope.go:117] "RemoveContainer" containerID="3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376"
Jul 13 20:14:15.275097 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.275078    2581 scope.go:117] "RemoveContainer" containerID="f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb"
Jul 13 20:14:15.275359 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:14:15.275341    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb\": container with ID starting with f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb not found: ID does not exist" containerID="f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb"
Jul 13 20:14:15.275418 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.275367    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb"} err="failed to get container status \"f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb\": rpc error: code = NotFound desc = could not find container \"f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb\": container with ID starting with f757e062796df7325423dbb8a92c4eadc5a6e3e88d1b824a06f217cf2fe5adbb not found: ID does not exist"
Jul 13 20:14:15.275418 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.275391    2581 scope.go:117] "RemoveContainer" containerID="3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376"
Jul 13 20:14:15.275610 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:14:15.275594    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376\": container with ID starting with 3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376 not found: ID does not exist" containerID="3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376"
Jul 13 20:14:15.275651 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.275615    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376"} err="failed to get container status \"3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376\": rpc error: code = NotFound desc = could not find container \"3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376\": container with ID starting with 3c61b87c54888a9f5611e631e164812719172e1c95d37e2e655786897b9fd376 not found: ID does not exist"
Jul 13 20:14:15.280174 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.280148    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"]
Jul 13 20:14:15.283934 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.283907    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5849bd8489-gtjrb"]
Jul 13 20:14:15.603446 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:15.603360    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" path="/var/lib/kubelet/pods/56a14432-596d-48e6-a8c5-ad5434d07ca2/volumes"
Jul 13 20:14:24.011315 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:24.011201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:14:24.030733 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:24.024211    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:14:45.290937 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.290864    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"]
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291194    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="storage-initializer"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291204    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="storage-initializer"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291211    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291217    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291223    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerName="storage-initializer"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291228    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerName="storage-initializer"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291234    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="storage-initializer"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291239    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="storage-initializer"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291256    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291262    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291267    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291310    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56a14432-596d-48e6-a8c5-ad5434d07ca2" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291319    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99cafc68-f538-44f0-98a9-1b317239a7a5" containerName="main"
Jul 13 20:14:45.291388 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.291326    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9943c2-679f-417e-b5bb-fe88aaf8a2a9" containerName="main"
Jul 13 20:14:45.295864 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.295843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.298838 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.298815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:14:45.298962 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.298910    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 13 20:14:45.299498 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.299479    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 20:14:45.320614 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.320580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"]
Jul 13 20:14:45.324399 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.324363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:45.413610 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d598899e-07fd-428c-a42a-c2ca2c13f571-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.413840 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d598899e-07fd-428c-a42a-c2ca2c13f571-tls-certs\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.413840 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.413840 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-model-cache\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.413840 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-dshm\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.413840 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-home\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.414045 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6q6\" (UniqueName: \"kubernetes.io/projected/d598899e-07fd-428c-a42a-c2ca2c13f571-kube-api-access-zx6q6\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.414045 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.413883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515158 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515158 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-model-cache\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515429 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-dshm\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515429 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-home\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515429 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6q6\" (UniqueName: \"kubernetes.io/projected/d598899e-07fd-428c-a42a-c2ca2c13f571-kube-api-access-zx6q6\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515429 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515429 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d598899e-07fd-428c-a42a-c2ca2c13f571-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515429 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d598899e-07fd-428c-a42a-c2ca2c13f571-tls-certs\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515745 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515838 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-model-cache\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515903 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-home\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.515903 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.515881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.516445 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.516423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d598899e-07fd-428c-a42a-c2ca2c13f571-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.517723 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.517699    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-dshm\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.517998 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.517980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d598899e-07fd-428c-a42a-c2ca2c13f571-tls-certs\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.522958 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.522936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6q6\" (UniqueName: \"kubernetes.io/projected/d598899e-07fd-428c-a42a-c2ca2c13f571-kube-api-access-zx6q6\") pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.606044 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.605966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:14:45.747454 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:45.747432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"]
Jul 13 20:14:45.749461 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:14:45.749432    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd598899e_07fd_428c_a42a_c2ca2c13f571.slice/crio-52ab02097494825abcf037ab2a3351454682a65e8d3848f30b4a215c19966b41 WatchSource:0}: Error finding container 52ab02097494825abcf037ab2a3351454682a65e8d3848f30b4a215c19966b41: Status 404 returned error can't find the container with id 52ab02097494825abcf037ab2a3351454682a65e8d3848f30b4a215c19966b41
Jul 13 20:14:46.412988 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:46.412950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerStarted","Data":"e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a"}
Jul 13 20:14:46.413371 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:14:46.413006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerStarted","Data":"52ab02097494825abcf037ab2a3351454682a65e8d3848f30b4a215c19966b41"}
Jul 13 20:15:00.482106 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:00.482075    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_lora-multiple-adapters-test-kserve-66df458ff4-5ntmf_d598899e-07fd-428c-a42a-c2ca2c13f571/storage-initializer/0.log"
Jul 13 20:15:00.482471 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:00.482118    2581 generic.go:356] "Generic (PLEG): container finished" podID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerID="e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a" exitCode=1
Jul 13 20:15:00.482471 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:00.482154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerDied","Data":"e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a"}
Jul 13 20:15:01.489135 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:01.489104    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_lora-multiple-adapters-test-kserve-66df458ff4-5ntmf_d598899e-07fd-428c-a42a-c2ca2c13f571/storage-initializer/0.log"
Jul 13 20:15:01.489611 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:01.489234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerStarted","Data":"dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59"}
Jul 13 20:15:05.512360 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:05.512324    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_lora-multiple-adapters-test-kserve-66df458ff4-5ntmf_d598899e-07fd-428c-a42a-c2ca2c13f571/storage-initializer/0.log"
Jul 13 20:15:05.512839 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:05.512375    2581 generic.go:356] "Generic (PLEG): container finished" podID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerID="dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59" exitCode=0
Jul 13 20:15:05.512839 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:05.512419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerDied","Data":"dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59"}
Jul 13 20:15:05.512839 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:05.512478    2581 scope.go:117] "RemoveContainer" containerID="e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a"
Jul 13 20:15:05.512987 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:05.512894    2581 scope.go:117] "RemoveContainer" containerID="e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a"
Jul 13 20:15:05.526361 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:15:05.526331    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_lora-multiple-adapters-test-kserve-66df458ff4-5ntmf_kserve-ci-e2e-test_d598899e-07fd-428c-a42a-c2ca2c13f571_0 in pod sandbox 52ab02097494825abcf037ab2a3351454682a65e8d3848f30b4a215c19966b41 from index: no such id: 'e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a'" containerID="e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a"
Jul 13 20:15:05.526463 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:15:05.526380    2581 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_lora-multiple-adapters-test-kserve-66df458ff4-5ntmf_kserve-ci-e2e-test_d598899e-07fd-428c-a42a-c2ca2c13f571_0 in pod sandbox 52ab02097494825abcf037ab2a3351454682a65e8d3848f30b4a215c19966b41 from index: no such id: 'e0e7c9bd26f9431e562730942b286731f129f8d913c33a570f4877c37a501a2a'; Skipping pod \"lora-multiple-adapters-test-kserve-66df458ff4-5ntmf_kserve-ci-e2e-test(d598899e-07fd-428c-a42a-c2ca2c13f571)\"" logger="UnhandledError"
Jul 13 20:15:06.519593 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:06.519558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerStarted","Data":"a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67"}
Jul 13 20:15:06.544053 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:06.543995    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podStartSLOduration=21.543973459 podStartE2EDuration="21.543973459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:06.542651887 +0000 UTC m=+5143.442877280" watchObservedRunningTime="2026-07-13 20:15:06.543973459 +0000 UTC m=+5143.444198854"
Jul 13 20:15:06.606563 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:06.606535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:15:07.876136 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:15:07.876086    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:15:08.530652 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:08.530623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:15:15.606583 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:15.606547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:15:15.608022 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:15.607991    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:15:25.607080 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:25.607042    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:15:35.606359 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:35.606312    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:15:44.570570 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:44.570537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:44.668973 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:44.668936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:45.606568 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:45.606520    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:15:55.606562 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:15:55.606519    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:16:05.607006 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:16:05.606952    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:16:15.606637 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:16:15.606548    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:16:25.616662 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:16:25.616634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:16:25.625426 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:16:25.625401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:16:46.621506 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:16:46.621472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:11.531950 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:17:11.531898    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:17:12.180301 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:17:12.180270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:17:38.741622 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:17:38.741586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:38.840348 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:17:38.840316    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:48.029108 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:17:48.029020    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"]
Jul 13 20:17:48.031481 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:17:48.031455    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs"]
Jul 13 20:17:49.603210 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:17:49.603176    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a106df84-5ae6-429d-b8b1-9c8491c9264d" path="/var/lib/kubelet/pods/a106df84-5ae6-429d-b8b1-9c8491c9264d/volumes"
Jul 13 20:18:21.395812 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:21.395753    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"]
Jul 13 20:18:21.420207 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:21.420180    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:25.168237 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:25.168204    2581 scope.go:117] "RemoveContainer" containerID="b7273bf6d4c67722c8dda38519399b8df1667a1aa780451140a9988708c4ec7a"
Jul 13 20:18:36.419696 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:36.419655    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" containerID="cri-o://a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67" gracePeriod=45
Jul 13 20:18:36.617921 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:36.617878    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 13 20:18:37.077522 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.077499    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:18:37.124385 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-kserve-provision-location\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.124385 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-dshm\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.124385 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-home\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.124385 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124388    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d598899e-07fd-428c-a42a-c2ca2c13f571-tls-certs\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.124732 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124419    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d598899e-07fd-428c-a42a-c2ca2c13f571-cabundle-cert\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.124732 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-model-cache\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.124732 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-tmp-dir\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.124732 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx6q6\" (UniqueName: \"kubernetes.io/projected/d598899e-07fd-428c-a42a-c2ca2c13f571-kube-api-access-zx6q6\") pod \"d598899e-07fd-428c-a42a-c2ca2c13f571\" (UID: \"d598899e-07fd-428c-a42a-c2ca2c13f571\") "
Jul 13 20:18:37.125002 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-home" (OuterVolumeSpecName: "home") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:18:37.125002 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d598899e-07fd-428c-a42a-c2ca2c13f571-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:18:37.125002 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.124896    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.125219 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.125195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-model-cache" (OuterVolumeSpecName: "model-cache") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:18:37.126748 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.126724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-dshm" (OuterVolumeSpecName: "dshm") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:18:37.126900 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.126828    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d598899e-07fd-428c-a42a-c2ca2c13f571-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:18:37.127047 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.127029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d598899e-07fd-428c-a42a-c2ca2c13f571-kube-api-access-zx6q6" (OuterVolumeSpecName: "kube-api-access-zx6q6") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "kube-api-access-zx6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:18:37.142235 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.142209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:18:37.199229 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.199192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d598899e-07fd-428c-a42a-c2ca2c13f571" (UID: "d598899e-07fd-428c-a42a-c2ca2c13f571"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:18:37.225287 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.225259    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d598899e-07fd-428c-a42a-c2ca2c13f571-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.225287 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.225281    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d598899e-07fd-428c-a42a-c2ca2c13f571-cabundle-cert\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.225287 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.225292    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.225496 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.225300    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.225496 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.225307    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6q6\" (UniqueName: \"kubernetes.io/projected/d598899e-07fd-428c-a42a-c2ca2c13f571-kube-api-access-zx6q6\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.225496 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.225316    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.225496 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.225325    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d598899e-07fd-428c-a42a-c2ca2c13f571-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:18:37.633682 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.633649    2581 generic.go:356] "Generic (PLEG): container finished" podID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerID="a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67" exitCode=0
Jul 13 20:18:37.634036 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.633703    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"
Jul 13 20:18:37.634036 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.633719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerDied","Data":"a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67"}
Jul 13 20:18:37.634036 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.633765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf" event={"ID":"d598899e-07fd-428c-a42a-c2ca2c13f571","Type":"ContainerDied","Data":"52ab02097494825abcf037ab2a3351454682a65e8d3848f30b4a215c19966b41"}
Jul 13 20:18:37.634036 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.633804    2581 scope.go:117] "RemoveContainer" containerID="a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67"
Jul 13 20:18:37.642417 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.642399    2581 scope.go:117] "RemoveContainer" containerID="dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59"
Jul 13 20:18:37.651748 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.651724    2581 scope.go:117] "RemoveContainer" containerID="a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67"
Jul 13 20:18:37.652027 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:18:37.652009    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67\": container with ID starting with a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67 not found: ID does not exist" containerID="a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67"
Jul 13 20:18:37.652090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.652037    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67"} err="failed to get container status \"a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67\": rpc error: code = NotFound desc = could not find container \"a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67\": container with ID starting with a5bdd0193410d06ffa91f69686e5a2da533e05addaa6782d951b6703891dca67 not found: ID does not exist"
Jul 13 20:18:37.652090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.652063    2581 scope.go:117] "RemoveContainer" containerID="dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59"
Jul 13 20:18:37.652179 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.652048    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"]
Jul 13 20:18:37.652310 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:18:37.652293    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59\": container with ID starting with dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59 not found: ID does not exist" containerID="dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59"
Jul 13 20:18:37.652357 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.652317    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59"} err="failed to get container status \"dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59\": rpc error: code = NotFound desc = could not find container \"dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59\": container with ID starting with dd7ac5535abfc258a3a662016be09a93aaace79e54472b21358016e06491fa59 not found: ID does not exist"
Jul 13 20:18:37.655337 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:37.655314    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-66df458ff4-5ntmf"]
Jul 13 20:18:39.602952 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:18:39.602921    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" path="/var/lib/kubelet/pods/d598899e-07fd-428c-a42a-c2ca2c13f571/volumes"
Jul 13 20:19:06.220535 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:06.220498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:15.181361 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:19:15.181277    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:19:15.818979 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:15.818946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:19:24.050797 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:24.050673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:19:24.064064 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:24.064046    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:19:39.201585 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201549    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"]
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201888    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="storage-initializer"
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201900    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="storage-initializer"
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201914    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="storage-initializer"
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201920    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="storage-initializer"
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201926    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main"
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201932    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main"
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.201999    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="main"
Jul 13 20:19:39.202023 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.202008    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d598899e-07fd-428c-a42a-c2ca2c13f571" containerName="storage-initializer"
Jul 13 20:19:39.206077 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.206061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.209747 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.209726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:19:39.210469 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.210451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 13 20:19:39.231502 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.231473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"]
Jul 13 20:19:39.249185 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.249158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:39.252383 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.252359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77143537-6740-4a40-8e7b-4f61f66bf333-tls-certs\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.252482 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.252387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-dshm\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.252482 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.252405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzj6\" (UniqueName: \"kubernetes.io/projected/77143537-6740-4a40-8e7b-4f61f66bf333-kube-api-access-djzj6\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.252482 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.252441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-kserve-provision-location\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.252604 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.252510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-model-cache\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.252604 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.252551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-home\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.252604 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.252569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-tmp-dir\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.352972 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.352936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-model-cache\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353168 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.352985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-home\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353168 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-tmp-dir\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353168 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77143537-6740-4a40-8e7b-4f61f66bf333-tls-certs\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353168 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-dshm\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353168 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzj6\" (UniqueName: \"kubernetes.io/projected/77143537-6740-4a40-8e7b-4f61f66bf333-kube-api-access-djzj6\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353168 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-kserve-provision-location\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353526 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-model-cache\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353526 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-home\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353526 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-tmp-dir\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.353526 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.353510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-kserve-provision-location\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.355583 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.355563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-dshm\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.355748 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.355731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77143537-6740-4a40-8e7b-4f61f66bf333-tls-certs\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.362438 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.362398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzj6\" (UniqueName: \"kubernetes.io/projected/77143537-6740-4a40-8e7b-4f61f66bf333-kube-api-access-djzj6\") pod \"tls-verification-test-kserve-759794d6b4-9m772\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.516210 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.516180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:39.529666 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.529642    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"]
Jul 13 20:19:39.533276 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.533257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.536090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.536071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-ztfp6\""
Jul 13 20:19:39.554301 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.554277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98666\" (UniqueName: \"kubernetes.io/projected/26354fd0-eb47-4635-a064-afdb7c363053-kube-api-access-98666\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.554455 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.554342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26354fd0-eb47-4635-a064-afdb7c363053-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.554455 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.554424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.554643 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.554479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.559092 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.559067    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"]
Jul 13 20:19:39.572138 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.572108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:39.655523 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.655484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98666\" (UniqueName: \"kubernetes.io/projected/26354fd0-eb47-4635-a064-afdb7c363053-kube-api-access-98666\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.655697 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.655547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26354fd0-eb47-4635-a064-afdb7c363053-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.655697 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.655583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.655697 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.655629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.656102 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.656075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.656102 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.656095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.658480 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.658453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26354fd0-eb47-4635-a064-afdb7c363053-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.673828 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.673777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98666\" (UniqueName: \"kubernetes.io/projected/26354fd0-eb47-4635-a064-afdb7c363053-kube-api-access-98666\") pod \"tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.675263 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.675202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"]
Jul 13 20:19:39.676691 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:19:39.676670    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77143537_6740_4a40_8e7b_4f61f66bf333.slice/crio-21788c2d1c608f045070e7fc8a1d02c8dad6f7329ff59afb8af197deb1af289d WatchSource:0}: Error finding container 21788c2d1c608f045070e7fc8a1d02c8dad6f7329ff59afb8af197deb1af289d: Status 404 returned error can't find the container with id 21788c2d1c608f045070e7fc8a1d02c8dad6f7329ff59afb8af197deb1af289d
Jul 13 20:19:39.846597 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.846510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:39.937454 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.937403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" event={"ID":"77143537-6740-4a40-8e7b-4f61f66bf333","Type":"ContainerStarted","Data":"ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0"}
Jul 13 20:19:39.938263 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:39.937466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" event={"ID":"77143537-6740-4a40-8e7b-4f61f66bf333","Type":"ContainerStarted","Data":"21788c2d1c608f045070e7fc8a1d02c8dad6f7329ff59afb8af197deb1af289d"}
Jul 13 20:19:40.002622 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:40.002583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"]
Jul 13 20:19:40.007280 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:19:40.007251    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26354fd0_eb47_4635_a064_afdb7c363053.slice/crio-7e9cce8ed7599afdc225f79a1dc3bf711179fc252bbd07de268b6cbfbf4892a8 WatchSource:0}: Error finding container 7e9cce8ed7599afdc225f79a1dc3bf711179fc252bbd07de268b6cbfbf4892a8: Status 404 returned error can't find the container with id 7e9cce8ed7599afdc225f79a1dc3bf711179fc252bbd07de268b6cbfbf4892a8
Jul 13 20:19:40.946632 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:40.946599    2581 generic.go:356] "Generic (PLEG): container finished" podID="26354fd0-eb47-4635-a064-afdb7c363053" containerID="c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144" exitCode=0
Jul 13 20:19:40.947141 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:40.946696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" event={"ID":"26354fd0-eb47-4635-a064-afdb7c363053","Type":"ContainerDied","Data":"c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144"}
Jul 13 20:19:40.947141 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:40.946759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" event={"ID":"26354fd0-eb47-4635-a064-afdb7c363053","Type":"ContainerStarted","Data":"7e9cce8ed7599afdc225f79a1dc3bf711179fc252bbd07de268b6cbfbf4892a8"}
Jul 13 20:19:41.953749 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:41.953711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" event={"ID":"26354fd0-eb47-4635-a064-afdb7c363053","Type":"ContainerStarted","Data":"a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913"}
Jul 13 20:19:41.954175 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:41.953880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:19:41.975238 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:41.975190    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" podStartSLOduration=2.171655912 podStartE2EDuration="2.975175908s" totalImagesPullingTime="0s" totalInitContainerRuntime="803.519996ms" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:41.973312207 +0000 UTC m=+5418.873537600" watchObservedRunningTime="2026-07-13 20:19:41.975175908 +0000 UTC m=+5418.875401300"
Jul 13 20:19:42.962604 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:42.962569    2581 generic.go:356] "Generic (PLEG): container finished" podID="77143537-6740-4a40-8e7b-4f61f66bf333" containerID="ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0" exitCode=0
Jul 13 20:19:42.963098 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:42.962581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" event={"ID":"77143537-6740-4a40-8e7b-4f61f66bf333","Type":"ContainerDied","Data":"ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0"}
Jul 13 20:19:43.969919 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:43.969885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" event={"ID":"77143537-6740-4a40-8e7b-4f61f66bf333","Type":"ContainerStarted","Data":"4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c"}
Jul 13 20:19:43.998735 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:43.998672    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podStartSLOduration=2.063575472 podStartE2EDuration="4.998652446s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.935076974s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:43.995548331 +0000 UTC m=+5420.895773723" watchObservedRunningTime="2026-07-13 20:19:43.998652446 +0000 UTC m=+5420.898877838"
Jul 13 20:19:44.517129 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:44.517092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:49.516816 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:49.516757    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:19:49.518426 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:49.518394    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:19:59.517421 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:19:59.517372    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:20:09.517524 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:09.517473    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:20:12.964452 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:12.964423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:20:19.517400 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:19.517354    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:20:29.517436 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:29.517387    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:20:39.517227 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:39.517187    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:20:42.157827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:42.157729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:49.516694 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:49.516594    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:20:59.517204 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:20:59.517158    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:21:09.517628 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:21:09.517582    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:21:18.819752 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:21:18.819709    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:21:19.481379 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:21:19.481349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:21:19.517215 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:21:19.517167    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:21:29.517204 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:21:29.517159    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:21:39.526850 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:21:39.526812    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:21:39.534938 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:21:39.534913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:22:15.514128 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:15.513865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:32.368074 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.368036    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"]
Jul 13 20:22:32.371501 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.371473    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"]
Jul 13 20:22:32.378030 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.377982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:32.380818 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:22:32.380768    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:22:32.380818 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:22:32.380818 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:22:32.380818 ip-10-0-131-203 kubenswrapper[2581]:  > containerID="a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913" cmd=["/bin/sleep","15"]
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.380838    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" message="" err=<
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]:  >
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:22:32.380857    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" podUID="26354fd0-eb47-4635-a064-afdb7c363053" containerName="main" containerID="cri-o://a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913"
Jul 13 20:22:32.381074 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.380896    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" podUID="26354fd0-eb47-4635-a064-afdb7c363053" containerName="main" containerID="cri-o://a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913" gracePeriod=60
Jul 13 20:22:32.438037 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.438010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:32.644719 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.644695    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:22:32.693974 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.693947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98666\" (UniqueName: \"kubernetes.io/projected/26354fd0-eb47-4635-a064-afdb7c363053-kube-api-access-98666\") pod \"26354fd0-eb47-4635-a064-afdb7c363053\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") "
Jul 13 20:22:32.694165 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.693986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-kserve-provision-location\") pod \"26354fd0-eb47-4635-a064-afdb7c363053\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") "
Jul 13 20:22:32.694250 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.694160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26354fd0-eb47-4635-a064-afdb7c363053-tls-certs\") pod \"26354fd0-eb47-4635-a064-afdb7c363053\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") "
Jul 13 20:22:32.694250 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.694219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-tokenizer-uds\") pod \"26354fd0-eb47-4635-a064-afdb7c363053\" (UID: \"26354fd0-eb47-4635-a064-afdb7c363053\") "
Jul 13 20:22:32.694642 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.694616    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "26354fd0-eb47-4635-a064-afdb7c363053" (UID: "26354fd0-eb47-4635-a064-afdb7c363053"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:22:32.694721 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.694677    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26354fd0-eb47-4635-a064-afdb7c363053" (UID: "26354fd0-eb47-4635-a064-afdb7c363053"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:22:32.696393 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.696368    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26354fd0-eb47-4635-a064-afdb7c363053-kube-api-access-98666" (OuterVolumeSpecName: "kube-api-access-98666") pod "26354fd0-eb47-4635-a064-afdb7c363053" (UID: "26354fd0-eb47-4635-a064-afdb7c363053"). InnerVolumeSpecName "kube-api-access-98666". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:22:32.696595 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.696575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26354fd0-eb47-4635-a064-afdb7c363053-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26354fd0-eb47-4635-a064-afdb7c363053" (UID: "26354fd0-eb47-4635-a064-afdb7c363053"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:22:32.795300 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.795268    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26354fd0-eb47-4635-a064-afdb7c363053-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:32.795300 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.795295    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-tokenizer-uds\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:32.795300 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.795304    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98666\" (UniqueName: \"kubernetes.io/projected/26354fd0-eb47-4635-a064-afdb7c363053-kube-api-access-98666\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:32.795512 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.795314    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26354fd0-eb47-4635-a064-afdb7c363053-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:32.844686 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.844656    2581 generic.go:356] "Generic (PLEG): container finished" podID="26354fd0-eb47-4635-a064-afdb7c363053" containerID="a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913" exitCode=0
Jul 13 20:22:32.844876 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.844709    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"
Jul 13 20:22:32.844876 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.844735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" event={"ID":"26354fd0-eb47-4635-a064-afdb7c363053","Type":"ContainerDied","Data":"a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913"}
Jul 13 20:22:32.844876 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.844811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8" event={"ID":"26354fd0-eb47-4635-a064-afdb7c363053","Type":"ContainerDied","Data":"7e9cce8ed7599afdc225f79a1dc3bf711179fc252bbd07de268b6cbfbf4892a8"}
Jul 13 20:22:32.844876 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.844823    2581 scope.go:117] "RemoveContainer" containerID="a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913"
Jul 13 20:22:32.854431 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.854415    2581 scope.go:117] "RemoveContainer" containerID="c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144"
Jul 13 20:22:32.862070 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.862055    2581 scope.go:117] "RemoveContainer" containerID="a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913"
Jul 13 20:22:32.862323 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:22:32.862304    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913\": container with ID starting with a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913 not found: ID does not exist" containerID="a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913"
Jul 13 20:22:32.862385 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.862331    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913"} err="failed to get container status \"a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913\": rpc error: code = NotFound desc = could not find container \"a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913\": container with ID starting with a3676325d84ab93486614c58ae33e07f0f7b2a682c66cfb6f7536ea7fac99913 not found: ID does not exist"
Jul 13 20:22:32.862385 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.862348    2581 scope.go:117] "RemoveContainer" containerID="c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144"
Jul 13 20:22:32.862581 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:22:32.862564    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144\": container with ID starting with c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144 not found: ID does not exist" containerID="c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144"
Jul 13 20:22:32.862621 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.862590    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144"} err="failed to get container status \"c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144\": rpc error: code = NotFound desc = could not find container \"c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144\": container with ID starting with c166ba8c6dc8248a49b706281e93eb36228b20080d60d0784798f394d1bbb144 not found: ID does not exist"
Jul 13 20:22:32.867386 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.867366    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"]
Jul 13 20:22:32.872619 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:32.872599    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5db688dfd5-ngfb8"]
Jul 13 20:22:33.604154 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:33.604125    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26354fd0-eb47-4635-a064-afdb7c363053" path="/var/lib/kubelet/pods/26354fd0-eb47-4635-a064-afdb7c363053/volumes"
Jul 13 20:22:47.395644 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:47.395599    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" containerID="cri-o://4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c" gracePeriod=45
Jul 13 20:22:47.527326 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:47.527286    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 13 20:22:48.144694 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.144665    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:22:48.231473 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-dshm\") pod \"77143537-6740-4a40-8e7b-4f61f66bf333\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") "
Jul 13 20:22:48.231663 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231496    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-model-cache\") pod \"77143537-6740-4a40-8e7b-4f61f66bf333\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") "
Jul 13 20:22:48.231663 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djzj6\" (UniqueName: \"kubernetes.io/projected/77143537-6740-4a40-8e7b-4f61f66bf333-kube-api-access-djzj6\") pod \"77143537-6740-4a40-8e7b-4f61f66bf333\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") "
Jul 13 20:22:48.231663 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77143537-6740-4a40-8e7b-4f61f66bf333-tls-certs\") pod \"77143537-6740-4a40-8e7b-4f61f66bf333\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") "
Jul 13 20:22:48.231663 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-kserve-provision-location\") pod \"77143537-6740-4a40-8e7b-4f61f66bf333\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") "
Jul 13 20:22:48.231663 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-home\") pod \"77143537-6740-4a40-8e7b-4f61f66bf333\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") "
Jul 13 20:22:48.231663 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-tmp-dir\") pod \"77143537-6740-4a40-8e7b-4f61f66bf333\" (UID: \"77143537-6740-4a40-8e7b-4f61f66bf333\") "
Jul 13 20:22:48.232026 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-model-cache" (OuterVolumeSpecName: "model-cache") pod "77143537-6740-4a40-8e7b-4f61f66bf333" (UID: "77143537-6740-4a40-8e7b-4f61f66bf333"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:22:48.232026 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.231973    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-model-cache\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:48.232481 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.232452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-home" (OuterVolumeSpecName: "home") pod "77143537-6740-4a40-8e7b-4f61f66bf333" (UID: "77143537-6740-4a40-8e7b-4f61f66bf333"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:22:48.233817 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.233766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-dshm" (OuterVolumeSpecName: "dshm") pod "77143537-6740-4a40-8e7b-4f61f66bf333" (UID: "77143537-6740-4a40-8e7b-4f61f66bf333"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:22:48.233933 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.233852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77143537-6740-4a40-8e7b-4f61f66bf333-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77143537-6740-4a40-8e7b-4f61f66bf333" (UID: "77143537-6740-4a40-8e7b-4f61f66bf333"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:22:48.234000 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.233941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77143537-6740-4a40-8e7b-4f61f66bf333-kube-api-access-djzj6" (OuterVolumeSpecName: "kube-api-access-djzj6") pod "77143537-6740-4a40-8e7b-4f61f66bf333" (UID: "77143537-6740-4a40-8e7b-4f61f66bf333"). InnerVolumeSpecName "kube-api-access-djzj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:22:48.245493 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.245471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77143537-6740-4a40-8e7b-4f61f66bf333" (UID: "77143537-6740-4a40-8e7b-4f61f66bf333"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:22:48.300761 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.300722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77143537-6740-4a40-8e7b-4f61f66bf333" (UID: "77143537-6740-4a40-8e7b-4f61f66bf333"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:22:48.332843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.332774    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-tmp-dir\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:48.332843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.332840    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-dshm\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:48.332843 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.332851    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzj6\" (UniqueName: \"kubernetes.io/projected/77143537-6740-4a40-8e7b-4f61f66bf333-kube-api-access-djzj6\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:48.333014 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.332860    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77143537-6740-4a40-8e7b-4f61f66bf333-tls-certs\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:48.333014 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.332871    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-kserve-provision-location\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:48.333014 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.332880    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77143537-6740-4a40-8e7b-4f61f66bf333-home\") on node \"ip-10-0-131-203.ec2.internal\" DevicePath \"\""
Jul 13 20:22:48.926660 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.926622    2581 generic.go:356] "Generic (PLEG): container finished" podID="77143537-6740-4a40-8e7b-4f61f66bf333" containerID="4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c" exitCode=0
Jul 13 20:22:48.927090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.926676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"
Jul 13 20:22:48.927090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.926709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" event={"ID":"77143537-6740-4a40-8e7b-4f61f66bf333","Type":"ContainerDied","Data":"4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c"}
Jul 13 20:22:48.927090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.926768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772" event={"ID":"77143537-6740-4a40-8e7b-4f61f66bf333","Type":"ContainerDied","Data":"21788c2d1c608f045070e7fc8a1d02c8dad6f7329ff59afb8af197deb1af289d"}
Jul 13 20:22:48.927090 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.926813    2581 scope.go:117] "RemoveContainer" containerID="4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c"
Jul 13 20:22:48.936487 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.936469    2581 scope.go:117] "RemoveContainer" containerID="ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0"
Jul 13 20:22:48.944470 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.944451    2581 scope.go:117] "RemoveContainer" containerID="4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c"
Jul 13 20:22:48.944732 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:22:48.944714    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c\": container with ID starting with 4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c not found: ID does not exist" containerID="4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c"
Jul 13 20:22:48.944773 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.944743    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c"} err="failed to get container status \"4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c\": rpc error: code = NotFound desc = could not find container \"4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c\": container with ID starting with 4602896ba4706eaf8a501f76ed7a173c0409b4c655d8154450682212c7504d8c not found: ID does not exist"
Jul 13 20:22:48.944773 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.944760    2581 scope.go:117] "RemoveContainer" containerID="ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0"
Jul 13 20:22:48.945005 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:22:48.944989    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0\": container with ID starting with ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0 not found: ID does not exist" containerID="ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0"
Jul 13 20:22:48.945043 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.945011    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0"} err="failed to get container status \"ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0\": rpc error: code = NotFound desc = could not find container \"ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0\": container with ID starting with ec316712e6ede9518801dd5c97020c2aa051fea127e1606f71e5265caa9f79f0 not found: ID does not exist"
Jul 13 20:22:48.948922 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.948900    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"]
Jul 13 20:22:48.952147 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:48.952125    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-759794d6b4-9m772"]
Jul 13 20:22:49.001169 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:49.001135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:49.604153 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:49.604116    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" path="/var/lib/kubelet/pods/77143537-6740-4a40-8e7b-4f61f66bf333/volumes"
Jul 13 20:22:55.567454 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:55.567422    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:55.668748 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:22:55.668716    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:00.413909 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:00.413869    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:12.254838 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:12.254810    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:12.292479 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:12.292437    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:12.292713 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:12.292694    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:12.324071 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:12.324038    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:12.324568 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:12.324549    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:14.201682 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:14.201652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:14.239765 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:14.239738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:14.239973 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:14.239956    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:14.265135 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:14.265110    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:14.265546 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:14.265525    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:16.117755 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:16.117729    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:16.166233 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:16.166190    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:16.166463 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:16.166446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:16.186996 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:16.186971    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:16.187264 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:16.187247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:18.065911 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:18.065862    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:18.086702 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:18.086650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:18.086954 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:18.086918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:18.105312 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:18.105272    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:18.105457 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:18.105440    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:19.873116 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:19.873081    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:19.890598 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:19.890567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:19.890955 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:19.890919    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:19.909115 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:19.909083    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:19.909504 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:19.909476    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:21.734976 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:21.734945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:21.754157 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:21.754133    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:21.754437 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:21.754421    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:21.769264 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:21.769239    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:21.769564 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:21.769547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:22.482877 ip-10-0-131-203 kubenswrapper[2581]: E0713 20:23:22.482831    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="2a700b56-6140-4f86-817f-877ba9bd95ce"
Jul 13 20:23:23.092445 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:23.092407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 13 20:23:23.481548 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:23.481518    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:23.513250 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:23.513224    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:23.513433 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:23.513384    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:23.584406 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:23.584380    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:23.584934 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:23.584916    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:25.352259 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:25.352232    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:25.370614 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:25.370589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:25.370895 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:25.370869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:25.386108 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:25.386083    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:25.386474 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:25.386454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:27.087482 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:27.087449    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:27.112847 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:27.112818    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:27.113162 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:27.113139    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:27.134891 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:27.134865    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:27.135195 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:27.135167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:28.783419 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:28.783382    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:28.802522 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:28.802487    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:28.802861 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:28.802825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:28.821046 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:28.821018    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:28.821238 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:28.821166    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:30.549248 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:30.549213    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:30.571307 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:30.571278    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:30.571603 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:30.571580    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:30.585634 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:30.585610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:30.586082 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:30.586059    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:32.296856 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:32.296826    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:32.318877 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:32.318845    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:32.319400 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:32.319371    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:32.333760 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:32.333731    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:32.334028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:32.333986    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:34.058869 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:34.058840    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:34.080780 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:34.080754    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:34.081037 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:34.081017    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:34.098053 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:34.098032    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:34.098501 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:34.098478    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:35.718986 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:35.718940    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:35.736723 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:35.736665    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:35.736928 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:35.736749    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:35.751858 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:35.751833    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:35.752277 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:35.752234    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:37.519131 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:37.519099    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:37.541606 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:37.541572    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:37.541767 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:37.541727    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:37.559717 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:37.559683    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:37.560074 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:37.560045    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:39.239960 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:39.239913    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb_dbcef857-215f-4fa4-8519-e5029d249889/main/0.log"
Jul 13 20:23:39.260772 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:39.260739    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/1.log"
Jul 13 20:23:39.260966 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:39.260899    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-r2bjb_9f1416fa-ab83-4554-ab21-2f2312113b39/istio-proxy/0.log"
Jul 13 20:23:39.274385 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:39.274364    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/1.log"
Jul 13 20:23:39.274545 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:39.274526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-c9txx_4b0ec9d9-4001-4475-b9a7-9632c3a8118f/istio-proxy/0.log"
Jul 13 20:23:41.484074 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:41.484034    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zprl6_f9546522-0517-4e48-98d3-5a452af57b10/discovery/0.log"
Jul 13 20:23:43.120329 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:43.120292    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zprl6_f9546522-0517-4e48-98d3-5a452af57b10/discovery/0.log"
Jul 13 20:23:44.816491 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:44.816405    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zprl6_f9546522-0517-4e48-98d3-5a452af57b10/discovery/0.log"
Jul 13 20:23:46.412103 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:46.412073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zprl6_f9546522-0517-4e48-98d3-5a452af57b10/discovery/0.log"
Jul 13 20:23:47.536781 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:47.536749    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:47.766239 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:47.766205    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:47.851290 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:47.851196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:49.018028 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:49.017998    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:49.297522 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:49.297424    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:49.334864 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:49.334837    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:50.535326 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:50.535297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:50.745638 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:50.745612    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:50.840600 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:50.840500    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:51.532737 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:51.532706    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:51.942334 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:51.942241    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:52.217142 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:52.217109    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:52.262281 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:52.262256    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:53.391335 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:53.391303    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:53.669553 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:53.669460    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:53.707971 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:53.707940    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:54.825479 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:54.825447    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:55.091273 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:55.091181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:55.128486 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:55.128454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:56.282710 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:56.282681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:56.571883 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:56.571773    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:56.612988 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:56.612955    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:57.734766 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:57.734739    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:58.004345 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:58.004250    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:58.048930 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:58.048901    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:23:59.189590 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:59.189559    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:23:59.460073 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:59.460043    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:23:59.501734 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:23:59.501704    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:00.615211 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:00.615157    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:00.889124 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:00.889044    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:00.946718 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:00.946685    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:02.096397 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:02.096363    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:02.308931 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:02.308906    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:02.388211 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:02.388130    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:03.537248 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:03.537222    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:03.815062 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:03.814978    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:03.879585 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:03.879560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:05.065258 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:05.065220    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:05.340381 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:05.340291    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:05.432682 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:05.432653    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:06.684840 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:06.684815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:06.959727 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:06.959687    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:06.994822 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:06.994779    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:08.106222 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:08.106191    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:08.379861 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:08.379755    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:08.412359 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:08.412322    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:09.578574 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:09.578540    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:09.828698 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:09.828609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:09.938309 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:09.938276    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:11.125546 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:11.125513    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:11.380079 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:11.379987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:11.429693 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:11.429665    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:12.607070 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:12.607040    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:12.879189 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:12.879096    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:12.932556 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:12.932529    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:14.125481 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:14.125450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:14.403284 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:14.403196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:14.552657 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:14.552628    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:15.680899 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:15.680867    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:15.936842 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:15.936681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:15.993326 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:15.993288    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:17.186248 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:17.186207    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:17.485870 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:17.485837    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:17.585739 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:17.585675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:18.743887 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:18.743838    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:19.021308 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:19.021229    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:19.074914 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:19.074886    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:20.174703 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:20.174670    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:20.449670 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:20.449589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:20.529907 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:20.529876    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:21.709869 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:21.709838    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:21.978233 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:21.978144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:22.147402 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:22.147370    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:23.439434 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:23.439387    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:23.715926 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:23.715891    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:23.765373 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:23.765337    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:24.081548 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:24.081385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:24:24.095091 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:24.095073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2s8k_cf537d94-8daa-47b7-9c7e-02e3c5e25f0c/ovn-acl-logging/0.log"
Jul 13 20:24:25.009309 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:25.009276    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:25.317508 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:25.317426    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:25.374173 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:25.374142    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:26.546707 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:26.546673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:26.816289 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:26.816207    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:26.879484 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:26.879451    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:28.009366 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:28.009328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:28.278936 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:28.278854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:28.326927 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:28.326901    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:29.526782 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:29.526745    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:29.789909 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:29.789830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:29.879683 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:29.879648    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:31.066761 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:31.066729    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:31.288676 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:31.288650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:31.455308 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:31.455283    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:32.821216 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:32.821181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:33.092410 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:33.092320    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:33.187895 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:33.187863    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:36.142537 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:36.142496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:36.352069 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:36.352029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:39.629006 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:39.628981    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q4wtd_c020b2f3-f25b-4d53-8e98-360ac0d1ede6/global-pull-secret-syncer/0.log"
Jul 13 20:24:39.776271 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:39.776240    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log"
Jul 13 20:24:40.189364 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:40.189326    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-203.ec2.internal_58f7c83942b70dd267414f6b25e3812d/haproxy/0.log"
Jul 13 20:24:43.587328 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:43.587288    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:43.765307 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:43.765279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4qmlg_f6e212d7-c36c-4e94-86b9-c34e88752fce/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:45.158071 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:45.158040    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-hphfv_03ff69ad-9d85-4022-a848-d302e2770e6c/manager/0.log"
Jul 13 20:24:45.473562 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:45.473535    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-cf52x_e7970fea-e3da-4538-88a3-dda7d391b794/manager/0.log"
Jul 13 20:24:45.600707 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:45.600670    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-mw8p8_75351e08-fdf3-40d9-a685-e3d447b251c6/manager/0.log"
Jul 13 20:24:48.466702 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:48.466665    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hb676_25e38820-4feb-490b-b66b-d72560a2ca58/node-exporter/0.log"
Jul 13 20:24:48.500754 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:48.500723    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hb676_25e38820-4feb-490b-b66b-d72560a2ca58/kube-rbac-proxy/0.log"
Jul 13 20:24:48.533665 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:48.533630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hb676_25e38820-4feb-490b-b66b-d72560a2ca58/init-textfile/0.log"
Jul 13 20:24:49.637447 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637413    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"]
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637708    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26354fd0-eb47-4635-a064-afdb7c363053" containerName="storage-initializer"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637718    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26354fd0-eb47-4635-a064-afdb7c363053" containerName="storage-initializer"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637734    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="storage-initializer"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637739    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="storage-initializer"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637749    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26354fd0-eb47-4635-a064-afdb7c363053" containerName="main"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637755    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26354fd0-eb47-4635-a064-afdb7c363053" containerName="main"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637766    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637771    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637834    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="77143537-6740-4a40-8e7b-4f61f66bf333" containerName="main"
Jul 13 20:24:49.637867 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.637842    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="26354fd0-eb47-4635-a064-afdb7c363053" containerName="main"
Jul 13 20:24:49.640833 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.640817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.644853 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.644835    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s6wcw\"/\"default-dockercfg-g9b4n\""
Jul 13 20:24:49.646026 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.646005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6wcw\"/\"openshift-service-ca.crt\""
Jul 13 20:24:49.646149 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.646041    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6wcw\"/\"kube-root-ca.crt\""
Jul 13 20:24:49.668610 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.668579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"]
Jul 13 20:24:49.814003 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.813969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-lib-modules\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.814202 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.814019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-sys\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.814202 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.814086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-podres\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.814202 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.814137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-proc\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.814202 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.814170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn827\" (UniqueName: \"kubernetes.io/projected/a387d880-efcc-4c6d-84ed-a0b92377064b-kube-api-access-zn827\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.915586 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-podres\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.915586 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-proc\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.915827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn827\" (UniqueName: \"kubernetes.io/projected/a387d880-efcc-4c6d-84ed-a0b92377064b-kube-api-access-zn827\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.915827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-proc\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.915827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-podres\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.915827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-lib-modules\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.915827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-sys\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.916004 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-sys\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.916004 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.915856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a387d880-efcc-4c6d-84ed-a0b92377064b-lib-modules\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.932290 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.932254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn827\" (UniqueName: \"kubernetes.io/projected/a387d880-efcc-4c6d-84ed-a0b92377064b-kube-api-access-zn827\") pod \"perf-node-gather-daemonset-wkgl6\" (UID: \"a387d880-efcc-4c6d-84ed-a0b92377064b\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:49.950114 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:49.950082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:50.098910 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:50.098879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"]
Jul 13 20:24:50.101724 ip-10-0-131-203 kubenswrapper[2581]: W0713 20:24:50.101698    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda387d880_efcc_4c6d_84ed_a0b92377064b.slice/crio-b00444e09a003f120fca057078936e92005e607a1c84f94f1e761bc8c02d2298 WatchSource:0}: Error finding container b00444e09a003f120fca057078936e92005e607a1c84f94f1e761bc8c02d2298: Status 404 returned error can't find the container with id b00444e09a003f120fca057078936e92005e607a1c84f94f1e761bc8c02d2298
Jul 13 20:24:50.526603 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:50.526567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6" event={"ID":"a387d880-efcc-4c6d-84ed-a0b92377064b","Type":"ContainerStarted","Data":"67bd152c340133eed943590eb55a9149445509bcd36fa66f1ffaaa1a0e115672"}
Jul 13 20:24:50.526827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:50.526618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6" event={"ID":"a387d880-efcc-4c6d-84ed-a0b92377064b","Type":"ContainerStarted","Data":"b00444e09a003f120fca057078936e92005e607a1c84f94f1e761bc8c02d2298"}
Jul 13 20:24:50.526827 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:50.526662    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:50.548580 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:50.548524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6" podStartSLOduration=1.548504879 podStartE2EDuration="1.548504879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:24:50.547519181 +0000 UTC m=+5727.447744572" watchObservedRunningTime="2026-07-13 20:24:50.548504879 +0000 UTC m=+5727.448730272"
Jul 13 20:24:53.929156 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:53.929124    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7f688_b19da82d-b0dc-40ac-ad54-b8acae858b1a/dns/0.log"
Jul 13 20:24:53.956434 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:53.956407    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7f688_b19da82d-b0dc-40ac-ad54-b8acae858b1a/kube-rbac-proxy/0.log"
Jul 13 20:24:54.137039 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:54.137011    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pd2tc_2e6cb61f-f3c0-4ef3-9a41-3143005c2fbf/dns-node-resolver/0.log"
Jul 13 20:24:54.892614 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:54.892574    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ql9cx_9e61056f-7549-4981-9e2f-885a7705b066/node-ca/0.log"
Jul 13 20:24:56.516285 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:56.516252    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zprl6_f9546522-0517-4e48-98d3-5a452af57b10/discovery/0.log"
Jul 13 20:24:56.543772 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:56.543750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-wkgl6"
Jul 13 20:24:57.213329 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:57.213297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wnkpx_1bea1125-1479-43d3-a72b-411ddb57cd13/serve-healthcheck-canary/0.log"
Jul 13 20:24:57.930440 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:57.930406    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7kmgq_5367e84b-17a8-4776-8254-f6edf0de77c9/kube-rbac-proxy/0.log"
Jul 13 20:24:57.950857 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:57.950830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7kmgq_5367e84b-17a8-4776-8254-f6edf0de77c9/exporter/0.log"
Jul 13 20:24:57.975348 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:24:57.975314    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7kmgq_5367e84b-17a8-4776-8254-f6edf0de77c9/extractor/0.log"
Jul 13 20:25:01.542520 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:25:01.542491    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zprl6_f9546522-0517-4e48-98d3-5a452af57b10/discovery/0.log"
Jul 13 20:25:02.391717 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:25:02.391652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6b69b6d547-w9wkj_bc4157cb-5aec-4bc3-b7bf-d33f146aefa7/manager/0.log"
Jul 13 20:25:02.490024 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:25:02.489977    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-c4ptz_41cd837e-531b-4956-b9b2-7ff100b732db/server/0.log"
Jul 13 20:25:03.366909 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:25:03.366874    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zprl6_f9546522-0517-4e48-98d3-5a452af57b10/discovery/0.log"
Jul 13 20:25:03.474850 ip-10-0-131-203 kubenswrapper[2581]: I0713 20:25:03.474817    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-gpt9r_6f512d55-c310-430a-b87f-cce0542b2ef4/manager/0.log"