Jul 06 20:49:17.421038 ip-10-0-140-38 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.861896    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865791    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865803    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865806    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865809    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865812    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865816    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865818    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:17.959183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865823    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:17.942934 ip-10-0-140-38 systemd[1]: Started Kubernetes Kubelet.
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865828    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865831    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865834    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865838    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865841    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865844    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865846    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865850    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865853    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865856    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865859    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865862    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865864    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865867    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865869    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865872    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865875    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865877    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:17.960366 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865880    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865883    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865885    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865888    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865891    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865893    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865896    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865899    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865901    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865904    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865906    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865911    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865914    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865917    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865919    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865922    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865925    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865928    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865931    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865933    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:17.961196 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865936    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865939    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865942    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865944    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865947    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865949    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865951    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865954    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865956    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865960    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865963    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865966    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865969    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865971    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865974    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865976    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865978    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865982    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865986    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865988    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:17.961997 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865991    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865994    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865996    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.865999    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866001    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866004    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866006    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866008    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866011    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866013    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866016    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866019    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866021    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866024    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866028    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866031    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866033    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866036    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866039    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866044    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:17.962649 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866046    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866048    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866051    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866054    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866057    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866060    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866062    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866065    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866069    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866071    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866074    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866077    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866079    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866803    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866820    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866823    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866826    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866829    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866832    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866834    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:17.963525 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866837    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866839    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866842    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866844    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866847    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866850    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866853    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866857    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866859    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866863    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866865    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866868    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866870    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866874    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866876    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866879    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866881    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866884    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866887    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866890    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:17.964346 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866892    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866895    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866898    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866901    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866904    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866906    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866909    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866911    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866914    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866916    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866919    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866921    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866924    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866926    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866929    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866932    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866935    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866938    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866940    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866943    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:17.965195 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866946    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866949    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866952    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866955    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866958    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866961    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866963    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866966    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866969    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866971    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866973    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866976    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866978    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866981    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866985    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866989    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866992    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866995    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:17.965858 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.866998    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867001    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867004    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867008    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867011    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867013    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867016    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867019    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867021    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867024    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867026    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867029    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867031    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867034    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867037    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867040    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867042    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867045    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867048    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867051    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867053    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:17.966540 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867055    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867061    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867063    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867066    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867069    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867071    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867076    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867080    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867099    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867104    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867108    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867111    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867114    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867529    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867535    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867539    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867542    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867545    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867548    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:17.967356 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867552    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867555    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867558    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867562    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867566    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867569    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867572    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867575    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867578    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867581    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867583    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867586    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867589    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867592    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867596    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867598    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867601    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867603    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867606    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:17.968183 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867608    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867611    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867613    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867616    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867618    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867621    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867623    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867626    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867628    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867631    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867633    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867635    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867638    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867640    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867643    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867646    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867649    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867652    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867655    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867658    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:17.971380 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867660    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867663    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867665    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867668    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867670    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867673    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867676    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867679    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867683    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867686    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867689    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867692    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867694    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867696    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867699    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867701    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867704    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867707    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867710    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867712    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:17.972834 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867729    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867732    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867734    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867737    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867740    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867743    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867746    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867748    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867751    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867754    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867757    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867760    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867762    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867765    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867768    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867770    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867773    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867775    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867778    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:17.979615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867782    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867784    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867787    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867789    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867792    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867794    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867797    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867800    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867802    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867805    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867807    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867810    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867813    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867816    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.867818    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867881    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867888    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867894    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867898    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867903    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867906    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 20:49:17.980636 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867910    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867915    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867919    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867922    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867926    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867929    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867932    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867935    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867939    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867942    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867945    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867949    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867954    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867957    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867960    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867963    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867967    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867971    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867975    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867978    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867981    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867984    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867987    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867990    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 20:49:17.981666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867993    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.867996    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868000    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868003    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868006    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868009    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868012    2578 flags.go:64] FLAG: --enable-server="true"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868015    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868019    2578 flags.go:64] FLAG: --event-burst="100"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868022    2578 flags.go:64] FLAG: --event-qps="50"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868025    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868028    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868031    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868035    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868038    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868041    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868046    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868049    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868052    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868056    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868059    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868062    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868064    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868068    2578 flags.go:64] FLAG: --feature-gates=""
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868071    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 20:49:17.982878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868074    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868077    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868080    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868083    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868086    2578 flags.go:64] FLAG: --help="false"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868089    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-38.ec2.internal"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868092    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868095    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868098    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868101    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868105    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868107    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868110    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868113    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868116    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868119    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868122    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868125    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868128    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868131    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868134    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868136    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868139    2578 flags.go:64] FLAG: --lock-file=""
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868143    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 20:49:17.984117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868146    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868149    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868155    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868158    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868161    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868164    2578 flags.go:64] FLAG: --logging-format="text"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868167    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868171    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868173    2578 flags.go:64] FLAG: --manifest-url=""
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868176    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868181    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868184    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868188    2578 flags.go:64] FLAG: --max-pods="110"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868191    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868194    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868197    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868200    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868203    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868206    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868209    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868216    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868219    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868222    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 20:49:18.142317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868225    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868228    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868234    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868237    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868240    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868243    2578 flags.go:64] FLAG: --port="10250"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868245    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868248    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0436b21905b18dbf5"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868253    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868256    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868260    2578 flags.go:64] FLAG: --register-node="true"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868263    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868267    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868270    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868273    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868276    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868279    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868282    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868285    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868288    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868291    2578 flags.go:64] FLAG: --runonce="false"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868294    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868297    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868300    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868302    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 20:49:18.165386 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868305    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868308    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868311    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868314    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868318    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868320    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868323    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868326    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868329    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868332    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868335    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868340    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868342    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868345    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868350    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868354    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868356    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868359    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868363    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868367    2578 flags.go:64] FLAG: --v="2"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868371    2578 flags.go:64] FLAG: --version="false"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868375    2578 flags.go:64] FLAG: --vmodule=""
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868379    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.868382    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868515    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:18.166796 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868520    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868523    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868525    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868528    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868530    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868533    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868536    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868538    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868541    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868544    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868546    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868549    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868552    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868554    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868557    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868560    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868563    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868565    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868568    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868570    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:18.167657 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868573    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868575    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868579    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868582    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868584    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868587    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868598    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868600    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868603    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868606    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868608    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868611    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868613    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868616    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868618    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868621    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868624    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868626    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868630    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:18.168496 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868634    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868636    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868639    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868642    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868644    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868647    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868649    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868652    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868655    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868657    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868660    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868662    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868665    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868667    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868670    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868674    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868676    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868678    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868681    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868684    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:18.169144 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868695    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868697    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868700    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868702    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868705    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868707    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868710    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868712    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868728    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868730    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868733    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868735    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868738    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868740    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868743    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868746    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868748    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868750    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868753    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868756    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:18.170237 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868758    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868760    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868764    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868768    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868771    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868774    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868776    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868780    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868783    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868787    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868790    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868795    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868798    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868807    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868810    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868813    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868815    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868818    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:18.170890 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.868821    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.869464    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.877981    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.877998    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.878012    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.878018    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878074    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878078    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878082    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878085    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878090    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878094    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878097    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878100    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:18.171478 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878103    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878107    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878110    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878113    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878116    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878119    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878122    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878125    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878128    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878131    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878134    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878137    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878141    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878144    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878147    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878150    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878156    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878158    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878161    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878165    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:18.171948 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878167    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878170    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878173    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878175    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878178    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878181    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878184    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878186    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878189    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878191    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878194    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878198    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878201    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878204    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878207    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878210    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878212    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878215    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878217    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:18.172588 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878220    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878223    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878226    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878228    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878231    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878235    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878238    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878240    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878243    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878246    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878249    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878253    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878256    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878279    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878282    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878286    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878289    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878292    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878295    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878298    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:18.173173 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878300    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878303    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878305    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878308    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878310    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878313    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878316    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878318    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878321    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878323    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878326    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878329    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878332    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878335    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878337    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878339    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878342    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878346    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878348    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:18.173815 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878351    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878353    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878357    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878359    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878362    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878365    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878367    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878370    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878373    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878375    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878378    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878380    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878383    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.878388    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878499    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:18.174409 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878504    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878507    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878510    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878512    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878515    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878518    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878520    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878523    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878526    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878528    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878531    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878535    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878537    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878541    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878545    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878549    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878552    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878555    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878557    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:18.174947 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878560    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878563    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878566    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878569    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878571    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878574    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878577    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878579    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878582    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878584    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878587    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878590    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878592    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878595    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878597    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878600    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878602    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878605    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878608    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:18.175910 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878611    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878613    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878615    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878618    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878620    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878623    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878626    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878628    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878631    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878635    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878637    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878640    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878642    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878645    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878649    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878651    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878654    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878657    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878660    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:18.176514 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878664    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878667    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878670    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878672    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878675    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878678    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878681    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878683    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878686    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878689    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878692    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878695    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878698    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878702    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878705    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878708    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878711    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878733    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878738    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878742    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:18.329098 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878747    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878750    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878754    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878757    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878759    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878762    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878764    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878768    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878771    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878773    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878776    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878779    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878781    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878784    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878786    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878788    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878791    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878794    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878797    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878799    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:18.329870 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:17.878802    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.878808    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.878914    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.882231    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.883813    2578 server.go:1019] "Starting client certificate rotation"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.883913    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.883948    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.886810    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.889051    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.903096    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.906612    2578 log.go:25] "Validated CRI v1 image API"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.907443    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 20:49:18.330520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.909971    2578 fs.go:135] Filesystem UUIDs: map[011417ca-b715-421d-a23d-dcd5a0ee4f30:/dev/nvme0n1p4 0f56bc24-3621-4006-96b7-65515378552d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 06 20:49:18.330969 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.909989    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 20:49:18.331045 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.916747    2578 manager.go:216] Machine: {Timestamp:2026-07-06 20:49:17.915406087 +0000 UTC m=+0.375167954 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090809 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a53303647bcd84660e966fc2e9f1c SystemUUID:ec2a5330-3647-bcd8-4660-e966fc2e9f1c BootID:25b4346c-fbbc-4733-b1c5-aeff6e84c7a1 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:24:e7:2b:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:24:e7:2b:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:7a:cb:ad:56:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 20:49:18.331045 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.916866    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 20:49:18.331045 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.917008    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 20:49:18.331045 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.918843    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:49:18.331045 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.922546    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 20:49:18.331045 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.922586    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 20:49:18.331045 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.923503    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.923517    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.923525    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.924189    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.927640    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.931678    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.931822    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.931851    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.931863    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.931884    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.931894    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.931906    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.932285    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.932302    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.934452    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.936590    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.936618    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937784    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937808    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937821    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937833    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937851    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937864    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937881    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937894    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937907    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937919    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937934    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.937946    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.938708    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.938732    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.942147    2578 server.go:1268] "Started kubelet"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.942953    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.943187    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.943299    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.944033    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zxnnz"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.944841    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:17.944972    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:17.944973    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.945056    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 20:49:18.331372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.945080    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:17.948889    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:17.950345    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-38.ec2.internal.18bfce7002e362b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-38.ec2.internal,UID:ip-10-0-140-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-38.ec2.internal,},FirstTimestamp:2026-07-06 20:49:17.942121145 +0000 UTC m=+0.401883016,LastTimestamp:2026-07-06 20:49:17.942121145 +0000 UTC m=+0.401883016,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-38.ec2.internal,}"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.951546    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.951561    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:17.951640    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.952428    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.952366    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.952559    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.952836    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.952846    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.953444    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zxnnz"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:17.966019    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:17.966093    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.967668    2578 factory.go:55] Registering systemd factory
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.967700    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.967988    2578 factory.go:153] Registering CRI-O factory
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.968006    2578 factory.go:223] Registration of the crio container factory successfully
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.968071    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.968103    2578 factory.go:103] Registering Raw factory
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.968120    2578 manager.go:1196] Started watching for new ooms in manager
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.968613    2578 manager.go:349] Starting recovery of all containers
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.977820    2578 manager.go:354] Recovery completed
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.983647    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.985624    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.985652    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.985663    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.986499    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.986507    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.986549    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.986840    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.987522    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:18.332502 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.987543    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.987555    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.988140    2578 policy_none.go:49] "None policy: Start"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.988152    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.988180    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:17.990032    2578 policy_none.go:47] "Start"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.022074    2578 manager.go:337] "Starting Device Plugin manager"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.022237    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.022248    2578 server.go:87] "Starting device plugin registration server"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.022458    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.022467    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.022677    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.022799    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.022810    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.023306    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.023371    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.125246    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.126600    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.126637    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.126652    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.126682    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.133275    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.133299    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-38.ec2.internal\": node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.152444    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.152571    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.153692    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.153725    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.153749    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.153785    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.156178    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.253130    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.254319    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal"]
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.254396    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.255416    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.255449    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.255466    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.257252    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.257384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.333611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.257414    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.258041    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.258046    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.258067    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.258084    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.258070    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.258162    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.259568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.259591    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.260224    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.260256    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.260275    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.311531    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-38.ec2.internal\" not found" node="ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.334743 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.316639    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-38.ec2.internal\" not found" node="ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.353548    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.354777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83d0de89030d1dc32eced03342c2d0df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal\" (UID: \"83d0de89030d1dc32eced03342c2d0df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.354806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83d0de89030d1dc32eced03342c2d0df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal\" (UID: \"83d0de89030d1dc32eced03342c2d0df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.354827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc314882863d165c307bf7eda7756faa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-38.ec2.internal\" (UID: \"dc314882863d165c307bf7eda7756faa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.454352    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.455570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83d0de89030d1dc32eced03342c2d0df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal\" (UID: \"83d0de89030d1dc32eced03342c2d0df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.455599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83d0de89030d1dc32eced03342c2d0df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal\" (UID: \"83d0de89030d1dc32eced03342c2d0df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.455618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc314882863d165c307bf7eda7756faa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-38.ec2.internal\" (UID: \"dc314882863d165c307bf7eda7756faa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.455649    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc314882863d165c307bf7eda7756faa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-38.ec2.internal\" (UID: \"dc314882863d165c307bf7eda7756faa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.455675    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83d0de89030d1dc32eced03342c2d0df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal\" (UID: \"83d0de89030d1dc32eced03342c2d0df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.471876 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.455680    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83d0de89030d1dc32eced03342c2d0df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal\" (UID: \"83d0de89030d1dc32eced03342c2d0df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.637504 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.560013    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.637504 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.618860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.637504 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.620046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:18.758788 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.660921    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.763305 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.761019    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.865746 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.862234    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:18.879162 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.878239    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:18.978208 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.883563    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 20:49:18.978208 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.883721    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:49:18.978208 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.883757    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:49:18.978208 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.957911    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:44:18 +0000 UTC" deadline="2028-01-10 12:06:26.802126084 +0000 UTC"
Jul 06 20:49:18.978208 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:18.957948    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13263h17m7.844183324s"
Jul 06 20:49:18.978208 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:18.963096    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:19.089398 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.064223    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:19.212229 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.164504    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:19.326992 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.265001    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:19.437262 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.365841    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:19.540720 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.466836    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:19.567431 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.567396    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-38.ec2.internal\" not found"
Jul 06 20:49:19.619002 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.618966    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:19.622702 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.622683    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:19.652226 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.652177    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:19.667187 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.667156    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:49:19.668182 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.668167    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal"
Jul 06 20:49:19.678077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.678056    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:49:19.759629 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:19.759594    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc314882863d165c307bf7eda7756faa.slice/crio-d1447b088ca647c33f57d29178b4f7403a3729f692d96cdd072ad7442ab74987 WatchSource:0}: Error finding container d1447b088ca647c33f57d29178b4f7403a3729f692d96cdd072ad7442ab74987: Status 404 returned error can't find the container with id d1447b088ca647c33f57d29178b4f7403a3729f692d96cdd072ad7442ab74987
Jul 06 20:49:19.759986 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:19.759965    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d0de89030d1dc32eced03342c2d0df.slice/crio-a72e608e3cd99eedf1e749f777e0a5c2ba472c52af2a3399cfdc67a1ab101cfd WatchSource:0}: Error finding container a72e608e3cd99eedf1e749f777e0a5c2ba472c52af2a3399cfdc67a1ab101cfd: Status 404 returned error can't find the container with id a72e608e3cd99eedf1e749f777e0a5c2ba472c52af2a3399cfdc67a1ab101cfd
Jul 06 20:49:19.764986 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.764969    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:49:19.767975 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.767949    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:19.932576 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.932483    2578 apiserver.go:52] "Watching apiserver"
Jul 06 20:49:19.942351 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.942319    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 20:49:19.943500 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.943475    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4jh4x","kube-system/konnectivity-agent-22bw2","kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vmv4k","openshift-image-registry/node-ca-8jxf9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal","openshift-multus/multus-n9hh5","openshift-multus/network-metrics-daemon-bg7sr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd","openshift-dns/node-resolver-xv2hw","openshift-multus/multus-additional-cni-plugins-8m4b7","openshift-network-diagnostics/network-check-target-rnnjh","openshift-network-operator/iptables-alerter-kknn9","openshift-ovn-kubernetes/ovnkube-node-ssq68"]
Jul 06 20:49:19.945844 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.945824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.947166 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.947142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:19.948549 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.948527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.949878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.949861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.951159 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.951097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:19.951275 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.951200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:49:19.951899 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.951881    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 20:49:19.952013 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.951996    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 20:49:19.952079 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.951996    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 20:49:19.952079 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.952065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 20:49:19.952079 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.952065    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 20:49:19.952241 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.952171    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 20:49:19.953666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.953641    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 20:49:19.954721 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.953777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 20:49:19.954939 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.954868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmbml\""
Jul 06 20:49:19.955079 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.955056    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 20:49:19.956107 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.955819    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 20:49:19.956107 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.955888    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:49:19.956258 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.956227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:19.956308 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.956296    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 20:49:19.956597 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.956574    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 20:49:19.957141 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.957025    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 20:49:19.957141 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.957075    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:19.957141 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.957115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 20:49:19.957315 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.957157    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 20:49:19.957315 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.957293    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 20:49:19.957447 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.957431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:19.957824 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.957806    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9c8zm\""
Jul 06 20:49:19.957970 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.957954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-phll7\""
Jul 06 20:49:19.958121 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.958105    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlrgk\""
Jul 06 20:49:19.958230 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.958214    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 20:49:19.958295 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.958278    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hq98n\""
Jul 06 20:49:19.959105 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.959086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:19.959204 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.959178    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:19.961064 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.961042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:19.962647 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.962632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:19.964128 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.964108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:19.965558 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.965540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:19.965635 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:19.965596    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:19.969913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.969880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-cni-multus\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970003 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.969917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-systemd\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970003 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.969946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-os-release\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970003 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.969970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-kubelet\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970121 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-env-overrides\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970121 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-var-lib-kubelet\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.970121 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-tuned\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.970121 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-k8s-cni-cncf-io\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970258 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-netns\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970258 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61661b85-2135-42c3-acb0-554ebb57828b-serviceca\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:19.970258 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bce170ed-ea9d-431e-9488-e79f29c66270-multus-daemon-config\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970258 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-cni-bin\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970258 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-kubelet\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970258 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-var-lib-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-host-slash\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-cni-netd\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-ovnkube-config\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-kubernetes\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-etc-kubernetes\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-slash\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-ovn\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-modprobe-d\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpnc\" (UniqueName: \"kubernetes.io/projected/61661b85-2135-42c3-acb0-554ebb57828b-kube-api-access-pbpnc\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-system-cni-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-cni-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-cnibin\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbe65930-f89d-457a-aa56-356423352e27-ovn-node-metrics-cert\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp69k\" (UniqueName: \"kubernetes.io/projected/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-kube-api-access-kp69k\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-log-socket\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-ovnkube-script-lib\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjmg\" (UniqueName: \"kubernetes.io/projected/dbe65930-f89d-457a-aa56-356423352e27-kube-api-access-4pjmg\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-conf-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysctl-conf\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-sys\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-lib-modules\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.970910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bce170ed-ea9d-431e-9488-e79f29c66270-cni-binary-copy\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-hostroot\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-run-netns\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61661b85-2135-42c3-acb0-554ebb57828b-host\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-etc-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.970984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysconfig\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-systemd-units\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-run\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-tmp\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjsf\" (UniqueName: \"kubernetes.io/projected/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-kube-api-access-npjsf\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-socket-dir-parent\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-multus-certs\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.971464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwb5\" (UniqueName: \"kubernetes.io/projected/bce170ed-ea9d-431e-9488-e79f29c66270-kube-api-access-2fwb5\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:19.971917 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-node-log\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.971917 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-cni-bin\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:19.971917 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysctl-d\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.971917 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-iptables-alerter-script\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:19.971917 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-systemd\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.971917 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-host\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.971917 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.971362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4jh\" (UniqueName: \"kubernetes.io/projected/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-kube-api-access-zl4jh\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:19.973040 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 20:49:19.973135 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973059    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 20:49:19.973135 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pqdzp\""
Jul 06 20:49:19.973254 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973176    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 20:49:19.973254 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973018    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 20:49:19.973467 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973452    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5cvm7\""
Jul 06 20:49:19.973535 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973453    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 20:49:19.973535 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973497    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 20:49:19.973680 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.973664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4j5j7\""
Jul 06 20:49:19.978877 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.978843    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 20:49:19.978877 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.978848    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 20:49:19.979540 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.978878    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 20:49:19.979540 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.979495    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pnmdp\""
Jul 06 20:49:19.992927 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:19.992904    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:49:20.025362 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.025336    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9skts"
Jul 06 20:49:20.035140 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.035119    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9skts"
Jul 06 20:49:20.054065 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.054033    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 20:49:20.072466 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-cni-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.072466 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-cnibin\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.072666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbe65930-f89d-457a-aa56-356423352e27-ovn-node-metrics-cert\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.072666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-cnibin\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.072666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-device-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.072666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a560bf6-4a79-4776-a235-fed8973ef3c6-konnectivity-ca\") pod \"konnectivity-agent-22bw2\" (UID: \"2a560bf6-4a79-4776-a235-fed8973ef3c6\") " pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:20.072666 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp69k\" (UniqueName: \"kubernetes.io/projected/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-kube-api-access-kp69k\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-log-socket\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-ovnkube-script-lib\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072690    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-cni-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-log-socket\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjmg\" (UniqueName: \"kubernetes.io/projected/dbe65930-f89d-457a-aa56-356423352e27-kube-api-access-4pjmg\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-conf-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.072913 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysctl-conf\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-sys\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-conf-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.072962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-lib-modules\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bce170ed-ea9d-431e-9488-e79f29c66270-cni-binary-copy\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-hostroot\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073020    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-run-netns\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073096    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-lib-modules\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61661b85-2135-42c3-acb0-554ebb57828b-host\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysctl-conf\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-etc-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073134    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-hostroot\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.073372 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073212    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysconfig\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-run-netns\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd34120d-7c4c-418c-a18a-502b063e4a74-dbus\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysconfig\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073293    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61661b85-2135-42c3-acb0-554ebb57828b-host\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-etc-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-sys\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-systemd-units\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-run\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073381    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-ovnkube-script-lib\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-tmp\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073444    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-run\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjsf\" (UniqueName: \"kubernetes.io/projected/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-kube-api-access-npjsf\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:20.074087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073444    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-systemd-units\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-system-cni-dir\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-socket-dir-parent\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-multus-certs\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwb5\" (UniqueName: \"kubernetes.io/projected/bce170ed-ea9d-431e-9488-e79f29c66270-kube-api-access-2fwb5\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bce170ed-ea9d-431e-9488-e79f29c66270-cni-binary-copy\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073662    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-multus-socket-dir-parent\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-node-log\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-cni-bin\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073705    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-multus-certs\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysctl-d\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-iptables-alerter-script\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:20.074826 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073777    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-node-log\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-systemd\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-host\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073831    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-cni-bin\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4jh\" (UniqueName: \"kubernetes.io/projected/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-kube-api-access-zl4jh\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8383939a-3749-4b54-b4c5-002e3df5779a-hosts-file\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073889    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-systemd\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073895    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-host\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8383939a-3749-4b54-b4c5-002e3df5779a-tmp-dir\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073901    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-sysctl-d\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrh7n\" (UniqueName: \"kubernetes.io/projected/1b9f1b31-50f7-47e6-940a-db563c935d88-kube-api-access-qrh7n\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.073984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-registration-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-cni-multus\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.075564 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-systemd\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074130    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-cni-multus\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074171    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-systemd\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chsbc\" (UniqueName: \"kubernetes.io/projected/99efd34a-e302-4d10-8c06-737e3708b051-kube-api-access-chsbc\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-os-release\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-kubelet\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-env-overrides\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-var-lib-kubelet\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074305    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-os-release\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074353    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-kubelet\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-tuned\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-k8s-cni-cncf-io\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074365    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-var-lib-kubelet\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.076313 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074354    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-iptables-alerter-script\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-netns\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074482    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-k8s-cni-cncf-io\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-run-netns\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61661b85-2135-42c3-acb0-554ebb57828b-serviceca\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-socket-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a560bf6-4a79-4776-a235-fed8973ef3c6-agent-certs\") pod \"konnectivity-agent-22bw2\" (UID: \"2a560bf6-4a79-4776-a235-fed8973ef3c6\") " pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bce170ed-ea9d-431e-9488-e79f29c66270-multus-daemon-config\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-os-release\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074712    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-env-overrides\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-cni-bin\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-kubelet\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-cni-bin\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-var-lib-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074825    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61661b85-2135-42c3-acb0-554ebb57828b-serviceca\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.074975    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-var-lib-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075027    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-host-var-lib-kubelet\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-sys-fs\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-host-slash\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075141    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-host-slash\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-openvswitch\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-cni-netd\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-ovnkube-config\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075277    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-cni-netd\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bce170ed-ea9d-431e-9488-e79f29c66270-multus-daemon-config\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-kubernetes\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.077565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd34120d-7c4c-418c-a18a-502b063e4a74-kubelet-config\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-etc-selinux\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-etc-kubernetes\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-slash\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-ovn\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-modprobe-d\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpnc\" (UniqueName: \"kubernetes.io/projected/61661b85-2135-42c3-acb0-554ebb57828b-kube-api-access-pbpnc\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt767\" (UniqueName: \"kubernetes.io/projected/8383939a-3749-4b54-b4c5-002e3df5779a-kube-api-access-dt767\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-cnibin\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-system-cni-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075719    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-host-slash\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075752    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dbe65930-f89d-457a-aa56-356423352e27-ovnkube-config\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.078028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-kubernetes\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075799    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-system-cni-dir\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075858    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bce170ed-ea9d-431e-9488-e79f29c66270-etc-kubernetes\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075858    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dbe65930-f89d-457a-aa56-356423352e27-run-ovn\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.075908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-modprobe-d\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.075929    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.076004    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs podName:edcbaf42-d2ca-4474-ac2b-cac85ffa7805 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:20.575985197 +0000 UTC m=+3.035747055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs") pod "network-metrics-daemon-bg7sr" (UID: "edcbaf42-d2ca-4474-ac2b-cac85ffa7805") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.077201    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-tmp\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.077248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbe65930-f89d-457a-aa56-356423352e27-ovn-node-metrics-cert\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.078440 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.077369    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-etc-tuned\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.096186 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.096142    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjmg\" (UniqueName: \"kubernetes.io/projected/dbe65930-f89d-457a-aa56-356423352e27-kube-api-access-4pjmg\") pod \"ovnkube-node-ssq68\" (UID: \"dbe65930-f89d-457a-aa56-356423352e27\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.097140 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.096600    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp69k\" (UniqueName: \"kubernetes.io/projected/5bec9cd0-90ac-460c-92a6-4c9264b8ca7d-kube-api-access-kp69k\") pod \"iptables-alerter-kknn9\" (UID: \"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d\") " pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:20.098163 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.097902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpnc\" (UniqueName: \"kubernetes.io/projected/61661b85-2135-42c3-acb0-554ebb57828b-kube-api-access-pbpnc\") pod \"node-ca-8jxf9\" (UID: \"61661b85-2135-42c3-acb0-554ebb57828b\") " pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:20.098163 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.097913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwb5\" (UniqueName: \"kubernetes.io/projected/bce170ed-ea9d-431e-9488-e79f29c66270-kube-api-access-2fwb5\") pod \"multus-n9hh5\" (UID: \"bce170ed-ea9d-431e-9488-e79f29c66270\") " pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.098163 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.098089    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjsf\" (UniqueName: \"kubernetes.io/projected/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-kube-api-access-npjsf\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:20.098457 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.098296    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4jh\" (UniqueName: \"kubernetes.io/projected/8f8d68f8-c9a2-43eb-9ffc-d01cb82870de-kube-api-access-zl4jh\") pod \"tuned-vmv4k\" (UID: \"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de\") " pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.159348 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.159287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal" event={"ID":"dc314882863d165c307bf7eda7756faa","Type":"ContainerStarted","Data":"d1447b088ca647c33f57d29178b4f7403a3729f692d96cdd072ad7442ab74987"}
Jul 06 20:49:20.161700 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.161641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal" event={"ID":"83d0de89030d1dc32eced03342c2d0df","Type":"ContainerStarted","Data":"a72e608e3cd99eedf1e749f777e0a5c2ba472c52af2a3399cfdc67a1ab101cfd"}
Jul 06 20:49:20.176490 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.176490 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-socket-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a560bf6-4a79-4776-a235-fed8973ef3c6-agent-certs\") pod \"konnectivity-agent-22bw2\" (UID: \"2a560bf6-4a79-4776-a235-fed8973ef3c6\") " pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-os-release\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-sys-fs\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd34120d-7c4c-418c-a18a-502b063e4a74-kubelet-config\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-etc-selinux\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt767\" (UniqueName: \"kubernetes.io/projected/8383939a-3749-4b54-b4c5-002e3df5779a-kube-api-access-dt767\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-cnibin\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.176756 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-device-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a560bf6-4a79-4776-a235-fed8973ef3c6-konnectivity-ca\") pod \"konnectivity-agent-22bw2\" (UID: \"2a560bf6-4a79-4776-a235-fed8973ef3c6\") " pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd34120d-7c4c-418c-a18a-502b063e4a74-dbus\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-system-cni-dir\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.176955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8383939a-3749-4b54-b4c5-002e3df5779a-hosts-file\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8383939a-3749-4b54-b4c5-002e3df5779a-tmp-dir\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrh7n\" (UniqueName: \"kubernetes.io/projected/1b9f1b31-50f7-47e6-940a-db563c935d88-kube-api-access-qrh7n\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177138 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177121    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-cnibin\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177187    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-device-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177195    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177229    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd34120d-7c4c-418c-a18a-502b063e4a74-dbus\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177255    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-socket-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177313    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd34120d-7c4c-418c-a18a-502b063e4a74-kubelet-config\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177314    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-system-cni-dir\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.177388    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177432    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8383939a-3749-4b54-b4c5-002e3df5779a-hosts-file\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.177458    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret podName:fd34120d-7c4c-418c-a18a-502b063e4a74 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:20.677436646 +0000 UTC m=+3.137198504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret") pod "global-pull-secret-syncer-4jh4x" (UID: "fd34120d-7c4c-418c-a18a-502b063e4a74") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:20.177672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-etc-selinux\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177705    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8383939a-3749-4b54-b4c5-002e3df5779a-tmp-dir\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177785    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a560bf6-4a79-4776-a235-fed8973ef3c6-konnectivity-ca\") pod \"konnectivity-agent-22bw2\" (UID: \"2a560bf6-4a79-4776-a235-fed8973ef3c6\") " pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177792    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-os-release\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-registration-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177889    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-sys-fs\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chsbc\" (UniqueName: \"kubernetes.io/projected/99efd34a-e302-4d10-8c06-737e3708b051-kube-api-access-chsbc\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-registration-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.177978    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99efd34a-e302-4d10-8c06-737e3708b051-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.178250 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.178048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b9f1b31-50f7-47e6-940a-db563c935d88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.178834 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.178615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b9f1b31-50f7-47e6-940a-db563c935d88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.180732 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.180701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a560bf6-4a79-4776-a235-fed8973ef3c6-agent-certs\") pod \"konnectivity-agent-22bw2\" (UID: \"2a560bf6-4a79-4776-a235-fed8973ef3c6\") " pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:20.188211 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.188145    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrh7n\" (UniqueName: \"kubernetes.io/projected/1b9f1b31-50f7-47e6-940a-db563c935d88-kube-api-access-qrh7n\") pod \"multus-additional-cni-plugins-8m4b7\" (UID: \"1b9f1b31-50f7-47e6-940a-db563c935d88\") " pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.189148 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.188852    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:20.189148 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.188880    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:20.189148 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.188895    2578 projected.go:196] Error preparing data for projected volume kube-api-access-49fqh for pod openshift-network-diagnostics/network-check-target-rnnjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:20.189148 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.188969    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh podName:1c681bbc-6740-4893-a051-c5f72d065516 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:20.688949869 +0000 UTC m=+3.148711731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-49fqh" (UniqueName: "kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh") pod "network-check-target-rnnjh" (UID: "1c681bbc-6740-4893-a051-c5f72d065516") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:20.191317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.191286    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt767\" (UniqueName: \"kubernetes.io/projected/8383939a-3749-4b54-b4c5-002e3df5779a-kube-api-access-dt767\") pod \"node-resolver-xv2hw\" (UID: \"8383939a-3749-4b54-b4c5-002e3df5779a\") " pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.193347 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.193316    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chsbc\" (UniqueName: \"kubernetes.io/projected/99efd34a-e302-4d10-8c06-737e3708b051-kube-api-access-chsbc\") pod \"aws-ebs-csi-driver-node-79xvd\" (UID: \"99efd34a-e302-4d10-8c06-737e3708b051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.257768 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.257718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9hh5"
Jul 06 20:49:20.263409 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.263384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kknn9"
Jul 06 20:49:20.270712 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.270666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:20.270845 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:20.270759    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce170ed_ea9d_431e_9488_e79f29c66270.slice/crio-a92fd5d8dca21071046a5e267fefcb909683253f882af702e92904148c9a2551 WatchSource:0}: Error finding container a92fd5d8dca21071046a5e267fefcb909683253f882af702e92904148c9a2551: Status 404 returned error can't find the container with id a92fd5d8dca21071046a5e267fefcb909683253f882af702e92904148c9a2551
Jul 06 20:49:20.275790 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.275742    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vmv4k"
Jul 06 20:49:20.279886 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:20.279816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bec9cd0_90ac_460c_92a6_4c9264b8ca7d.slice/crio-109f4a031eef38a430d01deebeab334def955a35788e6a7ac1f95a01ac0641b2 WatchSource:0}: Error finding container 109f4a031eef38a430d01deebeab334def955a35788e6a7ac1f95a01ac0641b2: Status 404 returned error can't find the container with id 109f4a031eef38a430d01deebeab334def955a35788e6a7ac1f95a01ac0641b2
Jul 06 20:49:20.281919 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.281897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jxf9"
Jul 06 20:49:20.283107 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:20.283083    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe65930_f89d_457a_aa56_356423352e27.slice/crio-aeae11210a9b3d95df14925979ee565a3f76288c211d7ff5a5a17429c2d28f69 WatchSource:0}: Error finding container aeae11210a9b3d95df14925979ee565a3f76288c211d7ff5a5a17429c2d28f69: Status 404 returned error can't find the container with id aeae11210a9b3d95df14925979ee565a3f76288c211d7ff5a5a17429c2d28f69
Jul 06 20:49:20.288397 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:20.288335    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8d68f8_c9a2_43eb_9ffc_d01cb82870de.slice/crio-1749b6d1cec5bdf7b36197e67b5c2d6201572422eba0ab8d32143736de2b94bb WatchSource:0}: Error finding container 1749b6d1cec5bdf7b36197e67b5c2d6201572422eba0ab8d32143736de2b94bb: Status 404 returned error can't find the container with id 1749b6d1cec5bdf7b36197e67b5c2d6201572422eba0ab8d32143736de2b94bb
Jul 06 20:49:20.289743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.289531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd"
Jul 06 20:49:20.296005 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.295978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:20.300459 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.300352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xv2hw"
Jul 06 20:49:20.305589 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.305566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m4b7"
Jul 06 20:49:20.329378 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:20.329085    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9f1b31_50f7_47e6_940a_db563c935d88.slice/crio-54fa4430379bf21ee2acaf646a51cc628691f0f7dc4bb8d0b6eea0b6874a2633 WatchSource:0}: Error finding container 54fa4430379bf21ee2acaf646a51cc628691f0f7dc4bb8d0b6eea0b6874a2633: Status 404 returned error can't find the container with id 54fa4430379bf21ee2acaf646a51cc628691f0f7dc4bb8d0b6eea0b6874a2633
Jul 06 20:49:20.580687 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.580588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:20.580930 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.580790    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:20.580930 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.580876    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs podName:edcbaf42-d2ca-4474-ac2b-cac85ffa7805 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:21.580853975 +0000 UTC m=+4.040616090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs") pod "network-metrics-daemon-bg7sr" (UID: "edcbaf42-d2ca-4474-ac2b-cac85ffa7805") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:20.681702 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.681649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:20.681899 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.681844    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:20.681970 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.681935    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret podName:fd34120d-7c4c-418c-a18a-502b063e4a74 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:21.681912154 +0000 UTC m=+4.141674014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret") pod "global-pull-secret-syncer-4jh4x" (UID: "fd34120d-7c4c-418c-a18a-502b063e4a74") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:20.782446 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:20.782395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:20.782654 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.782616    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:20.782654 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.782649    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:20.782791 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.782664    2578 projected.go:196] Error preparing data for projected volume kube-api-access-49fqh for pod openshift-network-diagnostics/network-check-target-rnnjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:20.782791 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:20.782765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh podName:1c681bbc-6740-4893-a051-c5f72d065516 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:21.782729598 +0000 UTC m=+4.242491455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-49fqh" (UniqueName: "kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh") pod "network-check-target-rnnjh" (UID: "1c681bbc-6740-4893-a051-c5f72d065516") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:21.038101 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.038012    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:44:20 +0000 UTC" deadline="2028-01-09 16:34:17.955993097 +0000 UTC"
Jul 06 20:49:21.038101 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.038053    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13243h44m56.917944743s"
Jul 06 20:49:21.058398 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.058366    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:21.154778 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.154724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:21.154990 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.154893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:21.164805 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.164769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xv2hw" event={"ID":"8383939a-3749-4b54-b4c5-002e3df5779a","Type":"ContainerStarted","Data":"72a622928e1360faf3a2fb52ba430919bacc2c9e2e1d6781d8b2851285cb94e7"}
Jul 06 20:49:21.166656 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.166620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd" event={"ID":"99efd34a-e302-4d10-8c06-737e3708b051","Type":"ContainerStarted","Data":"bcf20a5f2622b9356480e3d346dab8027550e9dcd102fb99ab53feb093155e02"}
Jul 06 20:49:21.169856 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.169828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jxf9" event={"ID":"61661b85-2135-42c3-acb0-554ebb57828b","Type":"ContainerStarted","Data":"03308320a1cc987413c13824309366c5ebc8684a050e4e55bd7be472e97edaa3"}
Jul 06 20:49:21.171425 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.171397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vmv4k" event={"ID":"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de","Type":"ContainerStarted","Data":"1749b6d1cec5bdf7b36197e67b5c2d6201572422eba0ab8d32143736de2b94bb"}
Jul 06 20:49:21.180040 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.180004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"aeae11210a9b3d95df14925979ee565a3f76288c211d7ff5a5a17429c2d28f69"}
Jul 06 20:49:21.183190 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.183154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kknn9" event={"ID":"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d","Type":"ContainerStarted","Data":"109f4a031eef38a430d01deebeab334def955a35788e6a7ac1f95a01ac0641b2"}
Jul 06 20:49:21.186272 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.186238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9hh5" event={"ID":"bce170ed-ea9d-431e-9488-e79f29c66270","Type":"ContainerStarted","Data":"a92fd5d8dca21071046a5e267fefcb909683253f882af702e92904148c9a2551"}
Jul 06 20:49:21.188865 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.188816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerStarted","Data":"54fa4430379bf21ee2acaf646a51cc628691f0f7dc4bb8d0b6eea0b6874a2633"}
Jul 06 20:49:21.193465 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.193435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-22bw2" event={"ID":"2a560bf6-4a79-4776-a235-fed8973ef3c6","Type":"ContainerStarted","Data":"1390ddaeba08595b9c5cf7cde593a9d75a3ed0c77501d14de1bfe6aa53a2ca18"}
Jul 06 20:49:21.590675 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.590634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:21.590899 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.590824    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:21.590980 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.590902    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs podName:edcbaf42-d2ca-4474-ac2b-cac85ffa7805 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:23.590881044 +0000 UTC m=+6.050642915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs") pod "network-metrics-daemon-bg7sr" (UID: "edcbaf42-d2ca-4474-ac2b-cac85ffa7805") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:21.691629 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.691573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:21.691861 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.691784    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:21.691861 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.691853    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret podName:fd34120d-7c4c-418c-a18a-502b063e4a74 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:23.691834638 +0000 UTC m=+6.151596494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret") pod "global-pull-secret-syncer-4jh4x" (UID: "fd34120d-7c4c-418c-a18a-502b063e4a74") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:21.792425 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:21.792379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:21.792673 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.792635    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:21.792673 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.792659    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:21.792673 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.792674    2578 projected.go:196] Error preparing data for projected volume kube-api-access-49fqh for pod openshift-network-diagnostics/network-check-target-rnnjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:21.792958 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:21.792740    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh podName:1c681bbc-6740-4893-a051-c5f72d065516 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:23.792718792 +0000 UTC m=+6.252480674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-49fqh" (UniqueName: "kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh") pod "network-check-target-rnnjh" (UID: "1c681bbc-6740-4893-a051-c5f72d065516") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:22.038622 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:22.038553    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:44:20 +0000 UTC" deadline="2027-12-17 13:00:00.415533253 +0000 UTC"
Jul 06 20:49:22.038622 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:22.038621    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12688h10m38.376917827s"
Jul 06 20:49:22.154043 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:22.154000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:22.154241 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:22.154150    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:22.154610 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:22.154588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:22.154740 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:22.154716    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:23.154634 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:23.154591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:23.155127 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.154767    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:23.212420 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:23.211886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal" event={"ID":"dc314882863d165c307bf7eda7756faa","Type":"ContainerStarted","Data":"461c9dcbd5fd3be18cdf781d24ea888c68c988443b56a549062a78f3814c60ff"}
Jul 06 20:49:23.610506 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:23.610470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:23.610705 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.610626    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:23.610705 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.610677    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs podName:edcbaf42-d2ca-4474-ac2b-cac85ffa7805 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:27.610664377 +0000 UTC m=+10.070426232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs") pod "network-metrics-daemon-bg7sr" (UID: "edcbaf42-d2ca-4474-ac2b-cac85ffa7805") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:23.712275 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:23.712232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:23.712446 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.712414    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:23.712504 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.712487    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret podName:fd34120d-7c4c-418c-a18a-502b063e4a74 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:27.712465765 +0000 UTC m=+10.172227667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret") pod "global-pull-secret-syncer-4jh4x" (UID: "fd34120d-7c4c-418c-a18a-502b063e4a74") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:23.812784 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:23.812665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:23.812979 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.812902    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:23.812979 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.812937    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:23.812979 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.812951    2578 projected.go:196] Error preparing data for projected volume kube-api-access-49fqh for pod openshift-network-diagnostics/network-check-target-rnnjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:23.813156 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:23.813021    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh podName:1c681bbc-6740-4893-a051-c5f72d065516 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:27.812999045 +0000 UTC m=+10.272760906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-49fqh" (UniqueName: "kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh") pod "network-check-target-rnnjh" (UID: "1c681bbc-6740-4893-a051-c5f72d065516") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:24.162613 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:24.162515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:24.163089 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:24.162716    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:24.163089 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:24.162876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:24.163233 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:24.163105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:24.220611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:24.220567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal" event={"ID":"83d0de89030d1dc32eced03342c2d0df","Type":"ContainerStarted","Data":"ff43258753ab5da8e6ea6eaad2d676f7aef081a0daf741f375e1a79bb3959a96"}
Jul 06 20:49:24.241871 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:24.240282    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal" podStartSLOduration=4.24025742 podStartE2EDuration="4.24025742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:23.231049457 +0000 UTC m=+5.690811339" watchObservedRunningTime="2026-07-06 20:49:24.24025742 +0000 UTC m=+6.700019291"
Jul 06 20:49:25.154248 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:25.154213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:25.154450 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:25.154360    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:26.154277 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:26.154185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:26.154747 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:26.154340    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:26.154882 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:26.154772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:26.154938 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:26.154908    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:27.154637 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:27.154602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:27.155079 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.154752    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:27.232904 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:27.232864    2578 generic.go:356] "Generic (PLEG): container finished" podID="83d0de89030d1dc32eced03342c2d0df" containerID="ff43258753ab5da8e6ea6eaad2d676f7aef081a0daf741f375e1a79bb3959a96" exitCode=0
Jul 06 20:49:27.233082 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:27.232920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal" event={"ID":"83d0de89030d1dc32eced03342c2d0df","Type":"ContainerDied","Data":"ff43258753ab5da8e6ea6eaad2d676f7aef081a0daf741f375e1a79bb3959a96"}
Jul 06 20:49:27.649017 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:27.648884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:27.649195 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.649038    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:27.649195 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.649105    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs podName:edcbaf42-d2ca-4474-ac2b-cac85ffa7805 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:35.64908549 +0000 UTC m=+18.108847359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs") pod "network-metrics-daemon-bg7sr" (UID: "edcbaf42-d2ca-4474-ac2b-cac85ffa7805") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:27.750268 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:27.750144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:27.750481 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.750278    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:27.750481 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.750353    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret podName:fd34120d-7c4c-418c-a18a-502b063e4a74 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:35.75033371 +0000 UTC m=+18.210095566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret") pod "global-pull-secret-syncer-4jh4x" (UID: "fd34120d-7c4c-418c-a18a-502b063e4a74") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:27.850798 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:27.850768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:27.850983 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.850935    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:27.850983 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.850956    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:27.850983 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.850968    2578 projected.go:196] Error preparing data for projected volume kube-api-access-49fqh for pod openshift-network-diagnostics/network-check-target-rnnjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:27.851135 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:27.851021    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh podName:1c681bbc-6740-4893-a051-c5f72d065516 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:35.85100233 +0000 UTC m=+18.310764199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-49fqh" (UniqueName: "kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh") pod "network-check-target-rnnjh" (UID: "1c681bbc-6740-4893-a051-c5f72d065516") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:28.155871 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:28.155219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:28.155871 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:28.155339    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:28.155871 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:28.155735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:28.155871 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:28.155825    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:29.154684 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:29.154640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:29.154896 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:29.154808    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:30.158410 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:30.158382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:30.158873 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:30.158501    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:30.158873 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:30.158553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:30.158873 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:30.158680    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:31.154354 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:31.154316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:31.154520 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:31.154449    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:32.158020 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:32.157933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:32.158396 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:32.158052    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:32.158396 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:32.158108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:32.158396 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:32.158237    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:33.154490 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:33.154455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:33.154657 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:33.154595    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:34.154038 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:34.154005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:34.154565 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:34.154143    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:34.154565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:34.154241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:34.154565 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:34.154345    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:35.154237 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:35.154192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:35.154674 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.154331    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:35.710986 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:35.710951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:35.711167 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.711075    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:35.711167 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.711153    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs podName:edcbaf42-d2ca-4474-ac2b-cac85ffa7805 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:51.711137538 +0000 UTC m=+34.170899392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs") pod "network-metrics-daemon-bg7sr" (UID: "edcbaf42-d2ca-4474-ac2b-cac85ffa7805") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:35.811611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:35.811531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:35.811790 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.811682    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:35.811790 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.811772    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret podName:fd34120d-7c4c-418c-a18a-502b063e4a74 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:51.811753981 +0000 UTC m=+34.271515839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret") pod "global-pull-secret-syncer-4jh4x" (UID: "fd34120d-7c4c-418c-a18a-502b063e4a74") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:35.912321 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:35.912278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:35.912502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.912449    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:35.912502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.912473    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:35.912502 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.912487    2578 projected.go:196] Error preparing data for projected volume kube-api-access-49fqh for pod openshift-network-diagnostics/network-check-target-rnnjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:35.912641 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:35.912540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh podName:1c681bbc-6740-4893-a051-c5f72d065516 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:51.912526279 +0000 UTC m=+34.372288133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-49fqh" (UniqueName: "kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh") pod "network-check-target-rnnjh" (UID: "1c681bbc-6740-4893-a051-c5f72d065516") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:36.154384 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:36.154294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:36.154384 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:36.154346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:36.154883 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:36.154458    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:36.154883 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:36.154589    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:37.154673 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:37.154633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:37.155070 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:37.154771    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:38.154579 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:38.154540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:38.154788 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:38.154640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:38.154788 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:38.154646    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:38.154788 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:38.154756    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:39.153961 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:39.153929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:39.154129 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:39.154063    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:40.157101 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.155938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:40.157101 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:40.156077    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:40.157101 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.156470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:40.157101 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:40.156559    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:40.260809 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.259685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"bd917ff94ac14f5384e75112cb4ff3483848a52412960689ad4dd79f864b9987"}
Jul 06 20:49:40.265565 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.265203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal" event={"ID":"83d0de89030d1dc32eced03342c2d0df","Type":"ContainerStarted","Data":"a64657ded01a7eac3b70f7fb98bd62f71650cf8356f1ba5e40b763c745a94f94"}
Jul 06 20:49:40.269492 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.269459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jxf9" event={"ID":"61661b85-2135-42c3-acb0-554ebb57828b","Type":"ContainerStarted","Data":"27ed4f579e90dc881b22ab035c68cd65573b56ea8fda1decbbd4e92649f800f4"}
Jul 06 20:49:40.272215 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.272183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vmv4k" event={"ID":"8f8d68f8-c9a2-43eb-9ffc-d01cb82870de","Type":"ContainerStarted","Data":"9390adcda55e48ab66087132f5bb83bd2f87740b80693eea09fad2fdd5c01044"}
Jul 06 20:49:40.285416 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.285349    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-38.ec2.internal" podStartSLOduration=20.285329156 podStartE2EDuration="20.285329156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:40.284354027 +0000 UTC m=+22.744115904" watchObservedRunningTime="2026-07-06 20:49:40.285329156 +0000 UTC m=+22.745091033"
Jul 06 20:49:40.306500 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:40.306442    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vmv4k" podStartSLOduration=2.578209704 podStartE2EDuration="22.306409409s" totalImagesPullingTime="19.728199705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:40.305419239 +0000 UTC m=+22.765181116" watchObservedRunningTime="2026-07-06 20:49:40.306409409 +0000 UTC m=+22.766171287"
Jul 06 20:49:41.155070 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.154880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:41.155301 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:41.155169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:41.275661 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.275625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xv2hw" event={"ID":"8383939a-3749-4b54-b4c5-002e3df5779a","Type":"ContainerStarted","Data":"b3ac89397a8cd0ee4843baf5c0414ecfb703b78ae22ce8df0bdc8bb4411a8b3c"}
Jul 06 20:49:41.277178 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.277151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd" event={"ID":"99efd34a-e302-4d10-8c06-737e3708b051","Type":"ContainerStarted","Data":"afe0765bbc3735bbbb1c8f0845b4289090374a35c985b1e48b0484418df381f9"}
Jul 06 20:49:41.282761 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.282731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"56dd3e54062cd5b2533c545bdd6f8703810e369f1e035076574e2316de48628f"}
Jul 06 20:49:41.282880 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.282853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"387bae2a716bf767ffb906e5a3a1c5151aec9b8733533a3d9de3586bdf51b44d"}
Jul 06 20:49:41.282938 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.282899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"2f7db5c88409dec8d79bdac3a61b83ac5d4eb2df9ccc985f7bdb8b917c5082e9"}
Jul 06 20:49:41.282988 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.282960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"199d4a2207ddadc56c74e5418759cc8d2adc8dee385e9260a79bec1493f4ce5a"}
Jul 06 20:49:41.283046 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.283032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"7f52531ae877d0299b5fa7154bd718c85069f493a12cd8affede5fe1c2440141"}
Jul 06 20:49:41.284171 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.284147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9hh5" event={"ID":"bce170ed-ea9d-431e-9488-e79f29c66270","Type":"ContainerStarted","Data":"e3e479af382849da64bf8aeaf4bbf51073de2e065ffe0006efccb79af4cbb69e"}
Jul 06 20:49:41.285682 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.285659    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b9f1b31-50f7-47e6-940a-db563c935d88" containerID="83f15248809f1b96d2ad831e71bde383f72863e76341045feae3c9a3379b670f" exitCode=0
Jul 06 20:49:41.285827 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.285750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerDied","Data":"83f15248809f1b96d2ad831e71bde383f72863e76341045feae3c9a3379b670f"}
Jul 06 20:49:41.287212 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.287193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-22bw2" event={"ID":"2a560bf6-4a79-4776-a235-fed8973ef3c6","Type":"ContainerStarted","Data":"0db53791982d3da81bee7b2129b05260ca14f995bf1ac92b940ef8f1e53e3db5"}
Jul 06 20:49:41.301627 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.301581    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8jxf9" podStartSLOduration=3.604636451 podStartE2EDuration="23.301562527s" totalImagesPullingTime="19.696926076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:40.326651936 +0000 UTC m=+22.786413841" watchObservedRunningTime="2026-07-06 20:49:41.301562527 +0000 UTC m=+23.761324386"
Jul 06 20:49:41.301925 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.301904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xv2hw" podStartSLOduration=3.608191739 podStartE2EDuration="23.30189525s" totalImagesPullingTime="19.693703511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:41.301733286 +0000 UTC m=+23.761495172" watchObservedRunningTime="2026-07-06 20:49:41.30189525 +0000 UTC m=+23.761657125"
Jul 06 20:49:41.322269 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.322221    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n9hh5" podStartSLOduration=3.452061747 podStartE2EDuration="23.322207176s" totalImagesPullingTime="19.870145429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:41.322024676 +0000 UTC m=+23.781786569" watchObservedRunningTime="2026-07-06 20:49:41.322207176 +0000 UTC m=+23.781969052"
Jul 06 20:49:41.336934 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.336888    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-22bw2" podStartSLOduration=3.656690665 podStartE2EDuration="23.336872852s" totalImagesPullingTime="19.680182187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:41.336492524 +0000 UTC m=+23.796254400" watchObservedRunningTime="2026-07-06 20:49:41.336872852 +0000 UTC m=+23.796634729"
Jul 06 20:49:41.470453 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.470352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:41.538040 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.538016    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 20:49:41.549835 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.549809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:41.657707 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.657670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:41.658321 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:41.658304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:42.037569 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.037453    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:49:41.538036771Z","UUID":"60a82aaa-fb0e-4848-b3a9-4764553a2ea6","Handler":null,"Name":"","Endpoint":""}
Jul 06 20:49:42.039494 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.039472    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 20:49:42.039672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.039517    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 20:49:42.154839 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.154806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:42.155042 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.154806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:42.155042 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:42.154950    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:42.155042 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:42.155026    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:42.292068 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.291976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kknn9" event={"ID":"5bec9cd0-90ac-460c-92a6-4c9264b8ca7d","Type":"ContainerStarted","Data":"087b34baf43d06407533a910c36637afc97fe944ce0c83129e0c401223f6d7e5"}
Jul 06 20:49:42.294596 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.294561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd" event={"ID":"99efd34a-e302-4d10-8c06-737e3708b051","Type":"ContainerStarted","Data":"d91d9c081466ed91443829c3a1e8ed403ee62e9c262b797dbc93a568dd0a4d2c"}
Jul 06 20:49:42.294902 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.294878    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:42.295353 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.295330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-22bw2"
Jul 06 20:49:42.330076 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.330020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kknn9" podStartSLOduration=4.614001022 podStartE2EDuration="24.330004568s" totalImagesPullingTime="19.716003546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:42.313040431 +0000 UTC m=+24.772802335" watchObservedRunningTime="2026-07-06 20:49:42.330004568 +0000 UTC m=+24.789766443"
Jul 06 20:49:42.401675 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.401634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:42.503192 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.503158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:42.604388 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.604298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:42.704332 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.704293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:42.804509 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.804317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:42.904620 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:42.904582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:43.154999 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:43.154912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:43.155169 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:43.155047    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:43.299523 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:43.299485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd" event={"ID":"99efd34a-e302-4d10-8c06-737e3708b051","Type":"ContainerStarted","Data":"b9ff8a327bae60f9f929867e2958bbc9f8abda30dfcfef639a225539a9902ca3"}
Jul 06 20:49:43.303607 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:43.303569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"b20e27abdb37fcbcc56b1485d8da496852e5a3309425bd161195d5da8046642d"}
Jul 06 20:49:44.154534 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:44.154499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:44.154819 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:44.154537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:44.154819 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:44.154637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:44.154819 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:44.154760    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:45.154750 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:45.154705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:45.155388 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:45.154841    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:46.154773 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.154559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:46.155373 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.154559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:46.155373 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:46.155213    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:46.155373 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:46.155342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:46.313016 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.312986    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b9f1b31-50f7-47e6-940a-db563c935d88" containerID="72d386b09c212ee87627ca0f5c95a39f198ee51ee866639690da2ee5c116d0dc" exitCode=0
Jul 06 20:49:46.313199 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.313086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerDied","Data":"72d386b09c212ee87627ca0f5c95a39f198ee51ee866639690da2ee5c116d0dc"}
Jul 06 20:49:46.316635 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.316604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" event={"ID":"dbe65930-f89d-457a-aa56-356423352e27","Type":"ContainerStarted","Data":"8f05b39cb876d89e8ad9d6a5edc0b0c3ac2c713386cb894aab57f808326da824"}
Jul 06 20:49:46.316903 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.316882    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:46.332520 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.332493    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:46.336171 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.336136    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79xvd" podStartSLOduration=6.425533198 podStartE2EDuration="28.336123333s" totalImagesPullingTime="21.910590135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:43.319898244 +0000 UTC m=+25.779660141" watchObservedRunningTime="2026-07-06 20:49:46.336123333 +0000 UTC m=+28.795885208"
Jul 06 20:49:46.365266 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:46.365213    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68" podStartSLOduration=8.584868611 podStartE2EDuration="28.365196324s" totalImagesPullingTime="19.780327713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:46.363498823 +0000 UTC m=+28.823260700" watchObservedRunningTime="2026-07-06 20:49:46.365196324 +0000 UTC m=+28.824958200"
Jul 06 20:49:47.154606 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.154577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:47.154766 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:47.154700    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:47.321538 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.321386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerStarted","Data":"84ee573a5af8f44337cef7cdedd37c88d4210a3b6433288fcededb218a310a45"}
Jul 06 20:49:47.321989 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.321969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:47.322048 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.322034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:47.337993 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.337969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:49:47.487712 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.487605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bg7sr"]
Jul 06 20:49:47.487927 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.487798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:47.487927 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:47.487913    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:47.488424 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.488400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4jh4x"]
Jul 06 20:49:47.488536 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.488526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:47.488633 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:47.488613    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:47.489018 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.488994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rnnjh"]
Jul 06 20:49:47.489122 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.489079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:47.489185 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:47.489139    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:47.618139 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.618106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:47.719672 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:47.719643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:48.325859 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:48.325829    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b9f1b31-50f7-47e6-940a-db563c935d88" containerID="84ee573a5af8f44337cef7cdedd37c88d4210a3b6433288fcededb218a310a45" exitCode=0
Jul 06 20:49:48.326224 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:48.325904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerDied","Data":"84ee573a5af8f44337cef7cdedd37c88d4210a3b6433288fcededb218a310a45"}
Jul 06 20:49:49.154089 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:49.154052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:49.154267 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:49.154097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:49.154267 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:49.154156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:49.154267 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:49.154170    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:49.154267 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:49.154236    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:49.154416 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:49.154305    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:49.331216 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:49.331127    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b9f1b31-50f7-47e6-940a-db563c935d88" containerID="e9d7df105d69b02ad91069fd0e42db92da40289363567781eb21044e1c26064d" exitCode=0
Jul 06 20:49:49.331216 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:49.331190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerDied","Data":"e9d7df105d69b02ad91069fd0e42db92da40289363567781eb21044e1c26064d"}
Jul 06 20:49:51.142962 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.142876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:51.154735 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.154691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:51.154907 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.154755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:51.154907 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.154822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:51.154907 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.154843    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jh4x" podUID="fd34120d-7c4c-418c-a18a-502b063e4a74"
Jul 06 20:49:51.155068 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.154921    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bg7sr" podUID="edcbaf42-d2ca-4474-ac2b-cac85ffa7805"
Jul 06 20:49:51.155068 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.155008    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rnnjh" podUID="1c681bbc-6740-4893-a051-c5f72d065516"
Jul 06 20:49:51.239494 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.239456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:51.728338 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.728307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:51.728519 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.728450    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:51.728569 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.728555    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs podName:edcbaf42-d2ca-4474-ac2b-cac85ffa7805 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:23.728538883 +0000 UTC m=+66.188300741 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs") pod "network-metrics-daemon-bg7sr" (UID: "edcbaf42-d2ca-4474-ac2b-cac85ffa7805") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:51.829590 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.829552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:51.829776 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.829671    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:51.829776 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.829748    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret podName:fd34120d-7c4c-418c-a18a-502b063e4a74 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:23.829713096 +0000 UTC m=+66.289474958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret") pod "global-pull-secret-syncer-4jh4x" (UID: "fd34120d-7c4c-418c-a18a-502b063e4a74") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:51.851602 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.851546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:51.930665 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.930627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:51.930855 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.930835    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:51.930926 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.930860    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:51.930926 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.930870    2578 projected.go:196] Error preparing data for projected volume kube-api-access-49fqh for pod openshift-network-diagnostics/network-check-target-rnnjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:51.931030 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:51.930924    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh podName:1c681bbc-6740-4893-a051-c5f72d065516 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:23.930909802 +0000 UTC m=+66.390671657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-49fqh" (UniqueName: "kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh") pod "network-check-target-rnnjh" (UID: "1c681bbc-6740-4893-a051-c5f72d065516") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:51.933177 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:51.933139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:52.855020 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.854945    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-38.ec2.internal" event="NodeReady"
Jul 06 20:49:52.855420 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.855104    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 20:49:52.907869 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.907837    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nt7zj"]
Jul 06 20:49:52.910704 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.910675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:52.911790 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.911767    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f99wv"]
Jul 06 20:49:52.913404 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.913382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 20:49:52.913708 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.913689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:52.913827 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.913757    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 20:49:52.914396 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.914238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpffn\""
Jul 06 20:49:52.916594 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.916571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zzls9\""
Jul 06 20:49:52.917704 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.917314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 20:49:52.917704 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.917364    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 20:49:52.918505 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.918456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 20:49:52.924802 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.924652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nt7zj"]
Jul 06 20:49:52.925411 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:52.925351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f99wv"]
Jul 06 20:49:53.025789 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.025758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:53.039354 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.039320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.039354 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.039354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab4d61a5-c075-4890-90e0-23ae50b1e952-cert\") pod \"ingress-canary-f99wv\" (UID: \"ab4d61a5-c075-4890-90e0-23ae50b1e952\") " pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:53.039568 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.039421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-config-volume\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.039568 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.039450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdqh\" (UniqueName: \"kubernetes.io/projected/ab4d61a5-c075-4890-90e0-23ae50b1e952-kube-api-access-lvdqh\") pod \"ingress-canary-f99wv\" (UID: \"ab4d61a5-c075-4890-90e0-23ae50b1e952\") " pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:53.039568 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.039479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-tmp-dir\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.039568 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.039492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5qw\" (UniqueName: \"kubernetes.io/projected/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-kube-api-access-xj5qw\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.140131 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.140131 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab4d61a5-c075-4890-90e0-23ae50b1e952-cert\") pod \"ingress-canary-f99wv\" (UID: \"ab4d61a5-c075-4890-90e0-23ae50b1e952\") " pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:53.140131 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-config-volume\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.140393 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdqh\" (UniqueName: \"kubernetes.io/projected/ab4d61a5-c075-4890-90e0-23ae50b1e952-kube-api-access-lvdqh\") pod \"ingress-canary-f99wv\" (UID: \"ab4d61a5-c075-4890-90e0-23ae50b1e952\") " pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:53.140393 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-tmp-dir\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.140393 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5qw\" (UniqueName: \"kubernetes.io/projected/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-kube-api-access-xj5qw\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.140393 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:53.140215    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:53.140393 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:53.140286    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls podName:8b0d5561-6a42-4921-920b-9ec87a6c4a4e nodeName:}" failed. No retries permitted until 2026-07-06 20:49:53.640267179 +0000 UTC m=+36.100029033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls") pod "dns-default-nt7zj" (UID: "8b0d5561-6a42-4921-920b-9ec87a6c4a4e") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:53.140958 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140758    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-config-volume\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.141071 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.140831    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-tmp-dir\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.145018 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.144995    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab4d61a5-c075-4890-90e0-23ae50b1e952-cert\") pod \"ingress-canary-f99wv\" (UID: \"ab4d61a5-c075-4890-90e0-23ae50b1e952\") " pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:53.153397 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.153370    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5qw\" (UniqueName: \"kubernetes.io/projected/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-kube-api-access-xj5qw\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.153530 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.153421    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdqh\" (UniqueName: \"kubernetes.io/projected/ab4d61a5-c075-4890-90e0-23ae50b1e952-kube-api-access-lvdqh\") pod \"ingress-canary-f99wv\" (UID: \"ab4d61a5-c075-4890-90e0-23ae50b1e952\") " pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:53.154079 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.154053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:49:53.154079 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.154063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:49:53.154221 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.154062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:49:53.157145 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.157123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:49:53.157245 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.157210    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hgfwv\""
Jul 06 20:49:53.157245 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.157121    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:49:53.157359 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.157243    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:49:53.157359 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.157290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf9wp\""
Jul 06 20:49:53.157570 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.157553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:49:53.226487 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.226449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:53.231290 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.231264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f99wv"
Jul 06 20:49:53.510281 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.510243    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f99wv"]
Jul 06 20:49:53.513263 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:49:53.513234    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4d61a5_c075_4890_90e0_23ae50b1e952.slice/crio-0a75315841b0472ec06f8a00ac85f71362d65aaf47eca236523804d084010de7 WatchSource:0}: Error finding container 0a75315841b0472ec06f8a00ac85f71362d65aaf47eca236523804d084010de7: Status 404 returned error can't find the container with id 0a75315841b0472ec06f8a00ac85f71362d65aaf47eca236523804d084010de7
Jul 06 20:49:53.643404 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:53.643374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:53.643572 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:53.643533    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:53.643613 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:53.643600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls podName:8b0d5561-6a42-4921-920b-9ec87a6c4a4e nodeName:}" failed. No retries permitted until 2026-07-06 20:49:54.643581884 +0000 UTC m=+37.103343746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls") pod "dns-default-nt7zj" (UID: "8b0d5561-6a42-4921-920b-9ec87a6c4a4e") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:54.345151 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:54.345109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f99wv" event={"ID":"ab4d61a5-c075-4890-90e0-23ae50b1e952","Type":"ContainerStarted","Data":"0a75315841b0472ec06f8a00ac85f71362d65aaf47eca236523804d084010de7"}
Jul 06 20:49:54.348281 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:54.348247    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b9f1b31-50f7-47e6-940a-db563c935d88" containerID="9b121d898a17a925140d672057d634c815d9c443cc42f47daf7b14a7fb3048d2" exitCode=0
Jul 06 20:49:54.348394 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:54.348285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerDied","Data":"9b121d898a17a925140d672057d634c815d9c443cc42f47daf7b14a7fb3048d2"}
Jul 06 20:49:54.650523 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:54.650424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:54.650697 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:54.650612    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:54.650776 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:54.650696    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls podName:8b0d5561-6a42-4921-920b-9ec87a6c4a4e nodeName:}" failed. No retries permitted until 2026-07-06 20:49:56.650671552 +0000 UTC m=+39.110433409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls") pod "dns-default-nt7zj" (UID: "8b0d5561-6a42-4921-920b-9ec87a6c4a4e") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:55.355104 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:55.354893    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b9f1b31-50f7-47e6-940a-db563c935d88" containerID="dcd81f26272061ca357e20e9d1af0516c702f6692788699fef57308810da50fe" exitCode=0
Jul 06 20:49:55.355104 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:55.354989    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerDied","Data":"dcd81f26272061ca357e20e9d1af0516c702f6692788699fef57308810da50fe"}
Jul 06 20:49:56.358980 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:56.358939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f99wv" event={"ID":"ab4d61a5-c075-4890-90e0-23ae50b1e952","Type":"ContainerStarted","Data":"2edc8a18d5560b31249beac688d3c2aa00bbcdb0dffac678598d0953b70920ce"}
Jul 06 20:49:56.362241 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:56.362205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" event={"ID":"1b9f1b31-50f7-47e6-940a-db563c935d88","Type":"ContainerStarted","Data":"7e2bb95740cd5adb11053f515287d3813f7a15307e58f5a01bcded4d34d50c62"}
Jul 06 20:49:56.384040 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:56.383994    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f99wv" podStartSLOduration=2.235452126 podStartE2EDuration="4.383976938s" totalImagesPullingTime="2.148524812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:56.382973829 +0000 UTC m=+38.842735727" watchObservedRunningTime="2026-07-06 20:49:56.383976938 +0000 UTC m=+38.843738814"
Jul 06 20:49:56.412816 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:56.412763    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8m4b7" podStartSLOduration=8.564409368 podStartE2EDuration="38.412745603s" totalImagesPullingTime="29.848336235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:56.410953727 +0000 UTC m=+38.870715603" watchObservedRunningTime="2026-07-06 20:49:56.412745603 +0000 UTC m=+38.872507476"
Jul 06 20:49:56.538650 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:56.538584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:56.610202 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:56.610093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:56.669978 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:56.669944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:49:56.670143 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:56.670117    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:56.670208 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:49:56.670192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls podName:8b0d5561-6a42-4921-920b-9ec87a6c4a4e nodeName:}" failed. No retries permitted until 2026-07-06 20:50:00.670172142 +0000 UTC m=+43.129934016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls") pod "dns-default-nt7zj" (UID: "8b0d5561-6a42-4921-920b-9ec87a6c4a4e") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:58.122567 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:58.122536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:58.196964 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:49:58.196931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:00.700654 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:00.700613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:50:00.701092 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:00.700764    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:50:00.701092 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:00.700823    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls podName:8b0d5561-6a42-4921-920b-9ec87a6c4a4e nodeName:}" failed. No retries permitted until 2026-07-06 20:50:08.700805242 +0000 UTC m=+51.160567097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls") pod "dns-default-nt7zj" (UID: "8b0d5561-6a42-4921-920b-9ec87a6c4a4e") : secret "dns-default-metrics-tls" not found
Jul 06 20:50:04.810384 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.810352    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"]
Jul 06 20:50:04.862890 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.862845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"]
Jul 06 20:50:04.863058 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.862945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:04.868851 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.868824    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 06 20:50:04.868989 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.868825    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vldl7\""
Jul 06 20:50:04.868989 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.868877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 06 20:50:04.869837 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.869629    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 06 20:50:04.870408 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:04.870388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:50:05.030711 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.030679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aca76d7d-a60d-4838-98ed-a851fb5b68a8-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.030711 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.030707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aca76d7d-a60d-4838-98ed-a851fb5b68a8-config\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.030931 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.030746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzh5\" (UniqueName: \"kubernetes.io/projected/aca76d7d-a60d-4838-98ed-a851fb5b68a8-kube-api-access-2fzh5\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.131268 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.131176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aca76d7d-a60d-4838-98ed-a851fb5b68a8-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.131268 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.131217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aca76d7d-a60d-4838-98ed-a851fb5b68a8-config\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.131268 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.131238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzh5\" (UniqueName: \"kubernetes.io/projected/aca76d7d-a60d-4838-98ed-a851fb5b68a8-kube-api-access-2fzh5\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.131864 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.131846    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aca76d7d-a60d-4838-98ed-a851fb5b68a8-config\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.134820 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.134803    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aca76d7d-a60d-4838-98ed-a851fb5b68a8-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.140999 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.140977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzh5\" (UniqueName: \"kubernetes.io/projected/aca76d7d-a60d-4838-98ed-a851fb5b68a8-kube-api-access-2fzh5\") pod \"service-ca-operator-5d46d9ffd4-87rqt\" (UID: \"aca76d7d-a60d-4838-98ed-a851fb5b68a8\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.172198 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.172151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"
Jul 06 20:50:05.179562 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.179534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:05.281357 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.281329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:05.322035 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.322004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt"]
Jul 06 20:50:05.325042 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:05.325004    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca76d7d_a60d_4838_98ed_a851fb5b68a8.slice/crio-7dcf26f87a792052805737d8ab9d800e6f81383ce5d567b1a2f39320af9b84ce WatchSource:0}: Error finding container 7dcf26f87a792052805737d8ab9d800e6f81383ce5d567b1a2f39320af9b84ce: Status 404 returned error can't find the container with id 7dcf26f87a792052805737d8ab9d800e6f81383ce5d567b1a2f39320af9b84ce
Jul 06 20:50:05.383289 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:05.383208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt" event={"ID":"aca76d7d-a60d-4838-98ed-a851fb5b68a8","Type":"ContainerStarted","Data":"7dcf26f87a792052805737d8ab9d800e6f81383ce5d567b1a2f39320af9b84ce"}
Jul 06 20:50:08.088396 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.088365    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qhlj7"]
Jul 06 20:50:08.105710 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.105682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhlj7"]
Jul 06 20:50:08.106019 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.105768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.108921 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.108900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 20:50:08.110200 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.110181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 20:50:08.110296 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.110199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 20:50:08.110816 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.110793    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 20:50:08.110816 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.110811    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mz8wb\""
Jul 06 20:50:08.112273 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.112253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:08.188060 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.188024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:08.261284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.261247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.261436 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.261298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.261436 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.261322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-crio-socket\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.261436 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.261368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxhz\" (UniqueName: \"kubernetes.io/projected/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-api-access-qgxhz\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.261540 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.261463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-data-volume\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.362697 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.362617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.362697 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.362661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.362697 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.362685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-crio-socket\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.362960 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.362709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxhz\" (UniqueName: \"kubernetes.io/projected/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-api-access-qgxhz\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.362960 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.362754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-data-volume\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.362960 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.362770    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:08.362960 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.362825    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.362960 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.362842    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:08.862822233 +0000 UTC m=+51.322584109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:08.362960 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.362891    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:08.862871114 +0000 UTC m=+51.322632969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.363196 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.363034    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-crio-socket\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.363196 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.363149    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-data-volume\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.373416 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.373382    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxhz\" (UniqueName: \"kubernetes.io/projected/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-api-access-qgxhz\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.392178 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.392152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt" event={"ID":"aca76d7d-a60d-4838-98ed-a851fb5b68a8","Type":"ContainerStarted","Data":"615be4b225bfc9b0ca6af198c696288a927035f9bb3b375b3716968fd784c809"}
Jul 06 20:50:08.413013 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.412963    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt" podStartSLOduration=2.365953167 podStartE2EDuration="4.412946452s" totalImagesPullingTime="2.046993285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:08.411523235 +0000 UTC m=+50.871285110" watchObservedRunningTime="2026-07-06 20:50:08.412946452 +0000 UTC m=+50.872708327"
Jul 06 20:50:08.767407 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.767315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:50:08.767560 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.767465    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:50:08.767560 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.767534    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls podName:8b0d5561-6a42-4921-920b-9ec87a6c4a4e nodeName:}" failed. No retries permitted until 2026-07-06 20:50:24.767516563 +0000 UTC m=+67.227278418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls") pod "dns-default-nt7zj" (UID: "8b0d5561-6a42-4921-920b-9ec87a6c4a4e") : secret "dns-default-metrics-tls" not found
Jul 06 20:50:08.844425 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.844392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:08.868068 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.868028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.868217 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.868162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:08.868217 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.868171    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.868324 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.868236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:09.868220725 +0000 UTC m=+52.327982579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.868324 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.868244    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:08.868324 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:08.868314    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:09.868297654 +0000 UTC m=+52.328059511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:08.944409 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:08.944375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:09.275481 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:09.275451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:09.357567 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:09.357532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:09.585612 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:09.585520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:09.678461 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:09.678434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:09.874606 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:09.874522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:09.874606 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:09.874565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:09.874815 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:09.874641    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:09.874815 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:09.874708    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:11.874692303 +0000 UTC m=+54.334454158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:09.874815 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:09.874707    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:09.874815 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:09.874763    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:11.874753527 +0000 UTC m=+54.334515381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:10.261763 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:10.261661    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xv2hw_8383939a-3749-4b54-b4c5-002e3df5779a/dns-node-resolver/0.log"
Jul 06 20:50:10.654101 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:10.654070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jxf9_61661b85-2135-42c3-acb0-554ebb57828b/node-ca/0.log"
Jul 06 20:50:11.158689 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.158656    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x4jt8"]
Jul 06 20:50:11.174120 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.174092    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x4jt8"]
Jul 06 20:50:11.174299 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.174202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.176954 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.176918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 06 20:50:11.176954 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.176947    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 06 20:50:11.177128 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.176947    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 06 20:50:11.178082 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.178065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 06 20:50:11.178196 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.178071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t4jpc\""
Jul 06 20:50:11.182746 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.182709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvq7j\" (UniqueName: \"kubernetes.io/projected/4d084aa5-8565-4d03-898e-44bdb8f479c3-kube-api-access-xvq7j\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.182850 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.182785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d084aa5-8565-4d03-898e-44bdb8f479c3-signing-key\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.182914 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.182897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d084aa5-8565-4d03-898e-44bdb8f479c3-signing-cabundle\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.211498 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.211469    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:11.283552 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.283520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvq7j\" (UniqueName: \"kubernetes.io/projected/4d084aa5-8565-4d03-898e-44bdb8f479c3-kube-api-access-xvq7j\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.283669 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.283576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d084aa5-8565-4d03-898e-44bdb8f479c3-signing-key\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.283669 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.283657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d084aa5-8565-4d03-898e-44bdb8f479c3-signing-cabundle\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.284312 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.284285    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4d084aa5-8565-4d03-898e-44bdb8f479c3-signing-cabundle\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.285976 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.285954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4d084aa5-8565-4d03-898e-44bdb8f479c3-signing-key\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.290619 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.290591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:11.292980 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.292953    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvq7j\" (UniqueName: \"kubernetes.io/projected/4d084aa5-8565-4d03-898e-44bdb8f479c3-kube-api-access-xvq7j\") pod \"service-ca-758f599c5-x4jt8\" (UID: \"4d084aa5-8565-4d03-898e-44bdb8f479c3\") " pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.483789 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.483675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-x4jt8"
Jul 06 20:50:11.606586 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.606546    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x4jt8"]
Jul 06 20:50:11.609313 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:11.609286    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d084aa5_8565_4d03_898e_44bdb8f479c3.slice/crio-21a5a3a99d33c79489b211b31594831bc0f880ab28753ba707dcadfbbfcf94ac WatchSource:0}: Error finding container 21a5a3a99d33c79489b211b31594831bc0f880ab28753ba707dcadfbbfcf94ac: Status 404 returned error can't find the container with id 21a5a3a99d33c79489b211b31594831bc0f880ab28753ba707dcadfbbfcf94ac
Jul 06 20:50:11.653772 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.653734    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f99wv_ab4d61a5-c075-4890-90e0-23ae50b1e952/serve-healthcheck-canary/0.log"
Jul 06 20:50:11.888547 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.888513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:11.888547 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:11.888552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:11.888978 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:11.888646    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:11.888978 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:11.888688    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:11.888978 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:11.888735    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:15.888701178 +0000 UTC m=+58.348463055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:11.888978 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:11.888752    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:15.888744942 +0000 UTC m=+58.348506796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:12.406550 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:12.406511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-x4jt8" event={"ID":"4d084aa5-8565-4d03-898e-44bdb8f479c3","Type":"ContainerStarted","Data":"f219eaded16b091bcf87e3d51864fd90a5e5a2dfc69a32ff3dfff022584fc1a3"}
Jul 06 20:50:12.406751 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:12.406567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-x4jt8" event={"ID":"4d084aa5-8565-4d03-898e-44bdb8f479c3","Type":"ContainerStarted","Data":"21a5a3a99d33c79489b211b31594831bc0f880ab28753ba707dcadfbbfcf94ac"}
Jul 06 20:50:12.437807 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:12.437754    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-x4jt8" podStartSLOduration=1.437737372 podStartE2EDuration="1.437737372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:12.437280246 +0000 UTC m=+54.897042122" watchObservedRunningTime="2026-07-06 20:50:12.437737372 +0000 UTC m=+54.897499239"
Jul 06 20:50:12.445392 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:12.445365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:13.220905 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:13.220830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:13.512423 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:13.512330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:13.667047 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:13.667005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:13.772989 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:13.772954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:13.879284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:13.879250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:13.974546 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:13.974509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:14.864428 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:14.864382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:15.272561 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:15.272527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:15.471236 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:15.471196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:15.645032 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:15.644941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:15.723852 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:15.723819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:15.875960 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:15.875927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:15.915267 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:15.915177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:15.915267 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:15.915220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:15.915473 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:15.915295    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:15.915473 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:15.915317    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:15.915473 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:15.915379    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:23.915362849 +0000 UTC m=+66.375124710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:15.915473 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:50:15.915427    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls podName:9c84bc98-fc5b-4b73-b610-e77a0dbda6d5 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:23.91541561 +0000 UTC m=+66.375177469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhlj7" (UID: "9c84bc98-fc5b-4b73-b610-e77a0dbda6d5") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:16.877347 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:16.877314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:16.979562 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:16.979532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:17.082751 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:17.082705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:17.182467 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:17.182380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:17.475405 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:17.475314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:17.671741 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:17.671698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:17.805852 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:17.805821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:18.073813 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:18.073741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:18.674432 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:18.674397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:19.176156 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:19.176125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:19.276410 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:19.276378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:19.342254 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:19.342228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssq68"
Jul 06 20:50:19.470345 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:19.470266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:19.865583 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:19.865549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:20.080922 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:20.080889    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:20.474280 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:20.474250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:21.071284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:21.071246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:21.169664 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:21.169625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:21.302264 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:21.302232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:21.680425 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:21.680393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:21.868439 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:21.868410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:22.065281 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:22.065246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:22.484421 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:22.484335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:22.707984 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:22.707942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:22.805103 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:22.805073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:23.073108 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.073026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:23.271024 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.270979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:23.486638 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.486559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:23.670466 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.670433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:23.778800 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.778759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:50:23.781974 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.781952    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:50:23.791785 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.791761    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcbaf42-d2ca-4474-ac2b-cac85ffa7805-metrics-certs\") pod \"network-metrics-daemon-bg7sr\" (UID: \"edcbaf42-d2ca-4474-ac2b-cac85ffa7805\") " pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:50:23.871906 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.871867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:23.879989 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.879962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:50:23.882995 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.882975    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:50:23.892734 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.892688    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd34120d-7c4c-418c-a18a-502b063e4a74-original-pull-secret\") pod \"global-pull-secret-syncer-4jh4x\" (UID: \"fd34120d-7c4c-418c-a18a-502b063e4a74\") " pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:50:23.980559 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.980525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:23.980559 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.980563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:23.980840 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.980599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:50:23.981131 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.981108    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:23.982925 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.982901    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c84bc98-fc5b-4b73-b610-e77a0dbda6d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhlj7\" (UID: \"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5\") " pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:23.984627 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.984609    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:50:23.994473 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:23.994451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:50:24.004291 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.004267    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fqh\" (UniqueName: \"kubernetes.io/projected/1c681bbc-6740-4893-a051-c5f72d065516-kube-api-access-49fqh\") pod \"network-check-target-rnnjh\" (UID: \"1c681bbc-6740-4893-a051-c5f72d065516\") " pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:50:24.017999 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.017979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mz8wb\""
Jul 06 20:50:24.025542 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.025516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhlj7"
Jul 06 20:50:24.073632 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.073594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jh4x"
Jul 06 20:50:24.083543 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.083513    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hgfwv\""
Jul 06 20:50:24.089983 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.089955    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf9wp\""
Jul 06 20:50:24.090925 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.090862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:50:24.097508 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.097464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bg7sr"
Jul 06 20:50:24.170366 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.168914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhlj7"]
Jul 06 20:50:24.226694 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.226656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4jh4x"]
Jul 06 20:50:24.231493 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:24.231462    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd34120d_7c4c_418c_a18a_502b063e4a74.slice/crio-251f0eddc49f1040eee09294074f9ffa69cff738446559d8f73e073ef7e1eb0d WatchSource:0}: Error finding container 251f0eddc49f1040eee09294074f9ffa69cff738446559d8f73e073ef7e1eb0d: Status 404 returned error can't find the container with id 251f0eddc49f1040eee09294074f9ffa69cff738446559d8f73e073ef7e1eb0d
Jul 06 20:50:24.263392 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.263371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rnnjh"]
Jul 06 20:50:24.265807 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:24.265779    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c681bbc_6740_4893_a051_c5f72d065516.slice/crio-f627530d18c250b5971b035883b1a4045ba251b6baa65b084056d165d5d9bad0 WatchSource:0}: Error finding container f627530d18c250b5971b035883b1a4045ba251b6baa65b084056d165d5d9bad0: Status 404 returned error can't find the container with id f627530d18c250b5971b035883b1a4045ba251b6baa65b084056d165d5d9bad0
Jul 06 20:50:24.278219 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.278192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bg7sr"]
Jul 06 20:50:24.281527 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:24.281471    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcbaf42_d2ca_4474_ac2b_cac85ffa7805.slice/crio-30c5d73a15c37021d12b7a2f63d44ac7f8809b75a8a92350618fbc75c0632df3 WatchSource:0}: Error finding container 30c5d73a15c37021d12b7a2f63d44ac7f8809b75a8a92350618fbc75c0632df3: Status 404 returned error can't find the container with id 30c5d73a15c37021d12b7a2f63d44ac7f8809b75a8a92350618fbc75c0632df3
Jul 06 20:50:24.437522 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.437460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rnnjh" event={"ID":"1c681bbc-6740-4893-a051-c5f72d065516","Type":"ContainerStarted","Data":"f627530d18c250b5971b035883b1a4045ba251b6baa65b084056d165d5d9bad0"}
Jul 06 20:50:24.438668 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.438641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4jh4x" event={"ID":"fd34120d-7c4c-418c-a18a-502b063e4a74","Type":"ContainerStarted","Data":"251f0eddc49f1040eee09294074f9ffa69cff738446559d8f73e073ef7e1eb0d"}
Jul 06 20:50:24.439872 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.439853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bg7sr" event={"ID":"edcbaf42-d2ca-4474-ac2b-cac85ffa7805","Type":"ContainerStarted","Data":"30c5d73a15c37021d12b7a2f63d44ac7f8809b75a8a92350618fbc75c0632df3"}
Jul 06 20:50:24.440836 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.440817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhlj7" event={"ID":"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5","Type":"ContainerStarted","Data":"2a2be4e719cdea6f283e5ea58a2e7836366b2e85bfb7b4331a49216da022ba43"}
Jul 06 20:50:24.695587 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.695495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:24.786053 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.786008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:50:24.788925 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.788889    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b0d5561-6a42-4921-920b-9ec87a6c4a4e-metrics-tls\") pod \"dns-default-nt7zj\" (UID: \"8b0d5561-6a42-4921-920b-9ec87a6c4a4e\") " pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:50:24.870456 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:24.870417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:25.027849 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:25.027807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpffn\""
Jul 06 20:50:25.035381 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:25.035084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:50:25.246751 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:25.246477    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nt7zj"]
Jul 06 20:50:25.246751 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:25.246667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:25.250852 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:25.250818    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0d5561_6a42_4921_920b_9ec87a6c4a4e.slice/crio-7a2c7253ff2fbf3103a0210529764cbb45341dcf2c06f1dd5e7893b1ab1a4317 WatchSource:0}: Error finding container 7a2c7253ff2fbf3103a0210529764cbb45341dcf2c06f1dd5e7893b1ab1a4317: Status 404 returned error can't find the container with id 7a2c7253ff2fbf3103a0210529764cbb45341dcf2c06f1dd5e7893b1ab1a4317
Jul 06 20:50:25.446705 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:25.446623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nt7zj" event={"ID":"8b0d5561-6a42-4921-920b-9ec87a6c4a4e","Type":"ContainerStarted","Data":"7a2c7253ff2fbf3103a0210529764cbb45341dcf2c06f1dd5e7893b1ab1a4317"}
Jul 06 20:50:32.383902 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.383869    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"]
Jul 06 20:50:32.385986 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.385943    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"]
Jul 06 20:50:32.386323 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.386211    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.388126 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.388107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.390931 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.390881    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 06 20:50:32.390931 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.390927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 06 20:50:32.391097 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.390887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 06 20:50:32.391097 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.391065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 20:50:32.391179 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.391105    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:50:32.392109 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.392087    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gs7h6\""
Jul 06 20:50:32.392332 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.392314    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 20:50:32.392332 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.392345    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 20:50:32.392541 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.392358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 20:50:32.392651 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.392635    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 20:50:32.392758 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.392677    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 20:50:32.401788 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.401767    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 20:50:32.411851 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.411819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"]
Jul 06 20:50:32.415429 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.415400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"]
Jul 06 20:50:32.440142 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.440307 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cdbec6-d168-401f-ad49-fa9ce50416c2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.440307 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.440307 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.440307 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.440507 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-ca\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.440507 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5w7\" (UniqueName: \"kubernetes.io/projected/72cdbec6-d168-401f-ad49-fa9ce50416c2-kube-api-access-sd5w7\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.440507 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpbt\" (UniqueName: \"kubernetes.io/projected/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-kube-api-access-vnpbt\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.440507 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.440437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-hub\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.470651 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.470620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nt7zj" event={"ID":"8b0d5561-6a42-4921-920b-9ec87a6c4a4e","Type":"ContainerStarted","Data":"8d958f37e7a0c5183adefd81a5e74510176b7b4159c96c3b5afc8d9e35475441"}
Jul 06 20:50:32.470946 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.470686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nt7zj" event={"ID":"8b0d5561-6a42-4921-920b-9ec87a6c4a4e","Type":"ContainerStarted","Data":"011e5e3848c00b905be09e9a1b0e228e8b2ebc8b759a79394657a2bc89243e1f"}
Jul 06 20:50:32.470946 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.470739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:50:32.473414 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.473388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bg7sr" event={"ID":"edcbaf42-d2ca-4474-ac2b-cac85ffa7805","Type":"ContainerStarted","Data":"536b1d263183b42bd5b014984f1a6b49e38f21990b7b05774114a44f98a7a8c0"}
Jul 06 20:50:32.473524 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.473456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bg7sr" event={"ID":"edcbaf42-d2ca-4474-ac2b-cac85ffa7805","Type":"ContainerStarted","Data":"c1d37f4554434f7b664f66e1f18119eca91f7a05a56c67605497c07e768bb3d5"}
Jul 06 20:50:32.475355 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.475332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhlj7" event={"ID":"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5","Type":"ContainerStarted","Data":"128359bf328f9582e7106d9ee11cb797841a2af28ae064ff574d2e97fa86adc7"}
Jul 06 20:50:32.476837 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.476814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rnnjh" event={"ID":"1c681bbc-6740-4893-a051-c5f72d065516","Type":"ContainerStarted","Data":"c9e26f852db6bb69e4f67fd46ff04d7380cdc1921b27cdc43d691132d7e47d6c"}
Jul 06 20:50:32.476920 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.476868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:50:32.478203 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.478182    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4jh4x" event={"ID":"fd34120d-7c4c-418c-a18a-502b063e4a74","Type":"ContainerStarted","Data":"a290e774c8d9731c8168353f6524e2a40e6aab76ad1d890b51377b2a2f2874d9"}
Jul 06 20:50:32.541870 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.541832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.541870 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.541873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cdbec6-d168-401f-ad49-fa9ce50416c2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.542135 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.541895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.542135 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.541913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.542135 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.541929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.542283 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.542253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-ca\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.542338 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.542296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5w7\" (UniqueName: \"kubernetes.io/projected/72cdbec6-d168-401f-ad49-fa9ce50416c2-kube-api-access-sd5w7\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.543385 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.542378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpbt\" (UniqueName: \"kubernetes.io/projected/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-kube-api-access-vnpbt\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.543385 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.542432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-hub\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.543385 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.542594    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.543385 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.543382    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cdbec6-d168-401f-ad49-fa9ce50416c2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.544763 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.544733    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.544983 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.544966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-ca\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.545140 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.545121    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.545188 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.545129    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.545287 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.545272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/72cdbec6-d168-401f-ad49-fa9ce50416c2-hub\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.583223 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.583176    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nt7zj" podStartSLOduration=34.116831077 podStartE2EDuration="40.583160032s" totalImagesPullingTime="6.466328955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:32.582300692 +0000 UTC m=+75.042062569" watchObservedRunningTime="2026-07-06 20:50:32.583160032 +0000 UTC m=+75.042921907"
Jul 06 20:50:32.605112 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.605081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5w7\" (UniqueName: \"kubernetes.io/projected/72cdbec6-d168-401f-ad49-fa9ce50416c2-kube-api-access-sd5w7\") pod \"cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p\" (UID: \"72cdbec6-d168-401f-ad49-fa9ce50416c2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.634817 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.634742    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpbt\" (UniqueName: \"kubernetes.io/projected/a839a4e5-8884-47ff-b9ad-67b6cfb4057a-kube-api-access-vnpbt\") pod \"kube-storage-version-migrator-operator-846bf668f7-2w478\" (UID: \"a839a4e5-8884-47ff-b9ad-67b6cfb4057a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.660154 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.660099    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rnnjh" podStartSLOduration=67.207048266 podStartE2EDuration="1m14.660083389s" totalImagesPullingTime="7.453035123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:32.650710823 +0000 UTC m=+75.110472698" watchObservedRunningTime="2026-07-06 20:50:32.660083389 +0000 UTC m=+75.119845264"
Jul 06 20:50:32.687227 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.687173    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bg7sr" podStartSLOduration=67.571662003 podStartE2EDuration="1m14.687160654s" totalImagesPullingTime="7.115498651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:32.681070967 +0000 UTC m=+75.140832842" watchObservedRunningTime="2026-07-06 20:50:32.687160654 +0000 UTC m=+75.146922527"
Jul 06 20:50:32.696888 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.696853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"
Jul 06 20:50:32.708117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.708058    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4jh4x" podStartSLOduration=67.220707031 podStartE2EDuration="1m14.708039758s" totalImagesPullingTime="7.487332727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:32.706566226 +0000 UTC m=+75.166328103" watchObservedRunningTime="2026-07-06 20:50:32.708039758 +0000 UTC m=+75.167801633"
Jul 06 20:50:32.714167 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.714136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"
Jul 06 20:50:32.731151 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.731121    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"]
Jul 06 20:50:32.733743 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.733706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.739479 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.739422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 20:50:32.743655 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.743353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 20:50:32.743655 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.743417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ldn5x\""
Jul 06 20:50:32.743940 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.743920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 20:50:32.744441 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.744422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:32.748348 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.748323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 20:50:32.760807 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.760773    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"]
Jul 06 20:50:32.831019 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.830980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:32.844873 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.844844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-registry-certificates\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.845020 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.844881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-image-registry-private-configuration\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.845020 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.844904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-bound-sa-token\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.845020 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.845003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-ca-trust-extracted\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.845129 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.845030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-trusted-ca\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.845129 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.845066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-installation-pull-secrets\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.845129 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.845098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-registry-tls\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.845129 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.845119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdpq\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-kube-api-access-cpdpq\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.867648 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.867623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478"]
Jul 06 20:50:32.870031 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:32.870000    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda839a4e5_8884_47ff_b9ad_67b6cfb4057a.slice/crio-4000cb6f30220cf38f63452e04d51f07c69d2ed77366ac0ff7cb3c1ba28ec87c WatchSource:0}: Error finding container 4000cb6f30220cf38f63452e04d51f07c69d2ed77366ac0ff7cb3c1ba28ec87c: Status 404 returned error can't find the container with id 4000cb6f30220cf38f63452e04d51f07c69d2ed77366ac0ff7cb3c1ba28ec87c
Jul 06 20:50:32.872162 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.872135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p"]
Jul 06 20:50:32.876453 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:32.876421    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72cdbec6_d168_401f_ad49_fa9ce50416c2.slice/crio-0c9380b2377da4bd4a745e3327a08283e2e950de734441e334b8595fc84fbe50 WatchSource:0}: Error finding container 0c9380b2377da4bd4a745e3327a08283e2e950de734441e334b8595fc84fbe50: Status 404 returned error can't find the container with id 0c9380b2377da4bd4a745e3327a08283e2e950de734441e334b8595fc84fbe50
Jul 06 20:50:32.945956 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.945864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-registry-certificates\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.945956 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.945911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-image-registry-private-configuration\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.946186 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.945976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-bound-sa-token\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.946186 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.946018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-ca-trust-extracted\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.946186 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.946035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-trusted-ca\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.946186 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.946073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-installation-pull-secrets\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.946186 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.946110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-registry-tls\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.946186 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.946145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdpq\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-kube-api-access-cpdpq\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.946486 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.946425    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-ca-trust-extracted\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.947065 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.947037    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-trusted-ca\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.947224 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.947196    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-registry-certificates\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.948265 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.948239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:32.948681 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.948654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-installation-pull-secrets\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.948896 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.948878    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-registry-tls\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.949015 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.948995    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-image-registry-private-configuration\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.962239 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.962210    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-bound-sa-token\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:32.962354 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:32.962336    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdpq\" (UniqueName: \"kubernetes.io/projected/716b91f7-c9b4-4bff-a1a7-2f89e43f8858-kube-api-access-cpdpq\") pod \"image-registry-c7f8c7bcf-nl7sm\" (UID: \"716b91f7-c9b4-4bff-a1a7-2f89e43f8858\") " pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:33.044279 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.044248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:33.045219 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.045194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:33.212458 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.212373    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"]
Jul 06 20:50:33.216708 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:33.216666    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716b91f7_c9b4_4bff_a1a7_2f89e43f8858.slice/crio-9f9bdd6dc0899b3227da4533ded769d1ada4d5f1add8f3ad2827b6b6d162e9bf WatchSource:0}: Error finding container 9f9bdd6dc0899b3227da4533ded769d1ada4d5f1add8f3ad2827b6b6d162e9bf: Status 404 returned error can't find the container with id 9f9bdd6dc0899b3227da4533ded769d1ada4d5f1add8f3ad2827b6b6d162e9bf
Jul 06 20:50:33.487647 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.487076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm" event={"ID":"716b91f7-c9b4-4bff-a1a7-2f89e43f8858","Type":"ContainerStarted","Data":"da68d997e3c3f3318c140c4d3be222af12d9c1ad3f4c1abd40cbc09e2553f72a"}
Jul 06 20:50:33.487647 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.487139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm" event={"ID":"716b91f7-c9b4-4bff-a1a7-2f89e43f8858","Type":"ContainerStarted","Data":"9f9bdd6dc0899b3227da4533ded769d1ada4d5f1add8f3ad2827b6b6d162e9bf"}
Jul 06 20:50:33.487647 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.487307    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:33.490284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.490255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p" event={"ID":"72cdbec6-d168-401f-ad49-fa9ce50416c2","Type":"ContainerStarted","Data":"0c9380b2377da4bd4a745e3327a08283e2e950de734441e334b8595fc84fbe50"}
Jul 06 20:50:33.492209 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.492133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478" event={"ID":"a839a4e5-8884-47ff-b9ad-67b6cfb4057a","Type":"ContainerStarted","Data":"4000cb6f30220cf38f63452e04d51f07c69d2ed77366ac0ff7cb3c1ba28ec87c"}
Jul 06 20:50:33.499217 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.497778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhlj7" event={"ID":"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5","Type":"ContainerStarted","Data":"4283075ebed886484c55d180019546bde1d9d604f76ba0f8c748d1f5ee199a85"}
Jul 06 20:50:33.523348 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:33.522208    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm" podStartSLOduration=1.522184768 podStartE2EDuration="1.522184768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:33.520057191 +0000 UTC m=+75.979819064" watchObservedRunningTime="2026-07-06 20:50:33.522184768 +0000 UTC m=+75.981946644"
Jul 06 20:50:38.519284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.519184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p" event={"ID":"72cdbec6-d168-401f-ad49-fa9ce50416c2","Type":"ContainerStarted","Data":"7057532f33a9370d72de23749397247d91c04277b90342e9c28f3fff47890f0a"}
Jul 06 20:50:38.519284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.519236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p" event={"ID":"72cdbec6-d168-401f-ad49-fa9ce50416c2","Type":"ContainerStarted","Data":"205096cafd93375f04a8557c83e87c904f081245c9d84b0b854a099ba3e69d93"}
Jul 06 20:50:38.519284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.519249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p" event={"ID":"72cdbec6-d168-401f-ad49-fa9ce50416c2","Type":"ContainerStarted","Data":"935d8315d6f3ad0c2799b16c1f6db42415057b7075a24d26f0f2fb6edc99ccb4"}
Jul 06 20:50:38.520767 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.520739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478" event={"ID":"a839a4e5-8884-47ff-b9ad-67b6cfb4057a","Type":"ContainerStarted","Data":"ab8cf1309244d842b3e3e714e7724403323ab30c14dfd7ce5e9e384f9819d8ec"}
Jul 06 20:50:38.522745 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.522700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhlj7" event={"ID":"9c84bc98-fc5b-4b73-b610-e77a0dbda6d5","Type":"ContainerStarted","Data":"e3d7f581c40ebf3b1c0d5c04d4c49431cbc7d62a8c02defbc771edef4bfc3ebc"}
Jul 06 20:50:38.540147 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.540098    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8bdb8b96-dzs5p" podStartSLOduration=1.479937955 podStartE2EDuration="6.540083441s" totalImagesPullingTime="5.060145486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:38.538946243 +0000 UTC m=+80.998708120" watchObservedRunningTime="2026-07-06 20:50:38.540083441 +0000 UTC m=+80.999845323"
Jul 06 20:50:38.563907 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.563847    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qhlj7" podStartSLOduration=16.965891498 podStartE2EDuration="30.563824924s" totalImagesPullingTime="13.597933426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:50:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:38.563065282 +0000 UTC m=+81.022827160" watchObservedRunningTime="2026-07-06 20:50:38.563824924 +0000 UTC m=+81.023586804"
Jul 06 20:50:38.582170 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.582118    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2w478" podStartSLOduration=1.5308501730000001 podStartE2EDuration="6.582100365s" totalImagesPullingTime="5.051250192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:38.580847235 +0000 UTC m=+81.040609112" watchObservedRunningTime="2026-07-06 20:50:38.582100365 +0000 UTC m=+81.041862291"
Jul 06 20:50:38.825456 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.825424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:38.922555 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:38.922512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.048098 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.048050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.130983 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.130900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.217408 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.217377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.319087 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.318980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.393791 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.393692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.496857 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.496826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.577342 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.577309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:39.902604 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:39.902572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:40.010760 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:40.010732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:40.102265 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:40.102232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:40.701878 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:40.701847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:40.900704 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:40.900668    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:41.070865 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:41.070832    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:41.467939 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:41.467854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:41.571156 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:41.571121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:41.746533 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:41.746450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:41.860341 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:41.860308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:42.142688 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:42.142644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:42.243578 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:42.243538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:42.501198 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:42.501124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nt7zj"
Jul 06 20:50:42.727011 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:42.726979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:42.824987 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:42.824955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:42.927420 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:42.927383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:44.191848 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.188800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:44.410408 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.410371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:44.549910 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.549882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:44.577646 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.577617    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f9b97"]
Jul 06 20:50:44.580738 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.580707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.587241 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.587223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 20:50:44.587382 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.587361    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 20:50:44.588611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.588369    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 20:50:44.588611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.588485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 20:50:44.588611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.588376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4rs94\""
Jul 06 20:50:44.588882 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.588701    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 20:50:44.588935 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.588918    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 20:50:44.638985 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.638943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f98r\" (UniqueName: \"kubernetes.io/projected/eadaa795-ad53-4038-a3c0-a53e938f4ad7-kube-api-access-6f98r\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639173 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639173 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639173 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-wtmp\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639173 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-tls\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639173 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-sys\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639173 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-textfile\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639365 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eadaa795-ad53-4038-a3c0-a53e938f4ad7-metrics-client-ca\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.639365 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.639217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-root\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.650397 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.650367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:44.739771 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f98r\" (UniqueName: \"kubernetes.io/projected/eadaa795-ad53-4038-a3c0-a53e938f4ad7-kube-api-access-6f98r\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.739771 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-wtmp\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-tls\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-sys\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-textfile\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eadaa795-ad53-4038-a3c0-a53e938f4ad7-metrics-client-ca\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-root\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.739965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-sys\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740424 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.740037    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-wtmp\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740424 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.740080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eadaa795-ad53-4038-a3c0-a53e938f4ad7-root\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740424 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.740289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-textfile\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740568 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.740509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eadaa795-ad53-4038-a3c0-a53e938f4ad7-metrics-client-ca\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.740658 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.740641    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-accelerators-collector-config\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.742492 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.742472    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.742579 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.742505    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eadaa795-ad53-4038-a3c0-a53e938f4ad7-node-exporter-tls\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.752273 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.752245    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f98r\" (UniqueName: \"kubernetes.io/projected/eadaa795-ad53-4038-a3c0-a53e938f4ad7-kube-api-access-6f98r\") pod \"node-exporter-f9b97\" (UID: \"eadaa795-ad53-4038-a3c0-a53e938f4ad7\") " pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.890445 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.890355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f9b97"
Jul 06 20:50:44.901066 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:44.901035    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadaa795_ad53_4038_a3c0_a53e938f4ad7.slice/crio-ae8c33d1d8d1a866289af1903ca412564ce39e24b122bfeb7abc6a4a66b73cee WatchSource:0}: Error finding container ae8c33d1d8d1a866289af1903ca412564ce39e24b122bfeb7abc6a4a66b73cee: Status 404 returned error can't find the container with id ae8c33d1d8d1a866289af1903ca412564ce39e24b122bfeb7abc6a4a66b73cee
Jul 06 20:50:44.957464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:44.957432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:45.062984 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:45.062954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:45.169351 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:45.169250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:45.466813 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:45.466738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:45.546558 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:45.546510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9b97" event={"ID":"eadaa795-ad53-4038-a3c0-a53e938f4ad7","Type":"ContainerStarted","Data":"ae8c33d1d8d1a866289af1903ca412564ce39e24b122bfeb7abc6a4a66b73cee"}
Jul 06 20:50:45.587077 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:45.586999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:45.698686 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:45.698659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:46.551668 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:46.551629    2578 generic.go:356] "Generic (PLEG): container finished" podID="eadaa795-ad53-4038-a3c0-a53e938f4ad7" containerID="f04c2d5e94a4ec30b6ffd56b251faeb896e1cd125e49cf7dc682aa07610be532" exitCode=0
Jul 06 20:50:46.552068 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:46.551685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9b97" event={"ID":"eadaa795-ad53-4038-a3c0-a53e938f4ad7","Type":"ContainerDied","Data":"f04c2d5e94a4ec30b6ffd56b251faeb896e1cd125e49cf7dc682aa07610be532"}
Jul 06 20:50:47.516300 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:47.516269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:47.557814 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:47.557783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9b97" event={"ID":"eadaa795-ad53-4038-a3c0-a53e938f4ad7","Type":"ContainerStarted","Data":"f048418fd0dc591375e821f416e4b81c7a0e6d5c6447b1c8a41f286924afdae9"}
Jul 06 20:50:47.558213 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:47.557834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9b97" event={"ID":"eadaa795-ad53-4038-a3c0-a53e938f4ad7","Type":"ContainerStarted","Data":"6d1b2f4ec4fc38f8851bcd042a5f8fa164e9658db9e9b172b8b604c4df2c2700"}
Jul 06 20:50:47.584093 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:47.584025    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f9b97" podStartSLOduration=2.722727233 podStartE2EDuration="3.584004984s" totalImagesPullingTime="794.180804ms" totalInitContainerRuntime="67.096947ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:47.582061642 +0000 UTC m=+90.041823519" watchObservedRunningTime="2026-07-06 20:50:47.584004984 +0000 UTC m=+90.043766863"
Jul 06 20:50:47.610197 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:47.610160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:47.721848 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:47.721815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:47.823096 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:47.823066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:48.819145 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:48.819116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:49.152116 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.152042    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"]
Jul 06 20:50:49.153114 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.153088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:49.153985 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.153973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.158911 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.158887    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 06 20:50:49.159029 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.158890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tx4jl\""
Jul 06 20:50:49.164857 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.164835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"]
Jul 06 20:50:49.264950 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.264912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:49.277992 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.277963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdds9\" (UniqueName: \"kubernetes.io/projected/2dd0c77d-4ae2-41fd-9dd7-176f59aab38e-kube-api-access-gdds9\") pod \"monitoring-plugin-d5dcff7cf-b2xxg\" (UID: \"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.278167 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.278005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dd0c77d-4ae2-41fd-9dd7-176f59aab38e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-b2xxg\" (UID: \"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.364682 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.364651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:49.378785 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.378752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdds9\" (UniqueName: \"kubernetes.io/projected/2dd0c77d-4ae2-41fd-9dd7-176f59aab38e-kube-api-access-gdds9\") pod \"monitoring-plugin-d5dcff7cf-b2xxg\" (UID: \"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.378941 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.378802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dd0c77d-4ae2-41fd-9dd7-176f59aab38e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-b2xxg\" (UID: \"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.381381 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.381345    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dd0c77d-4ae2-41fd-9dd7-176f59aab38e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-b2xxg\" (UID: \"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.394980 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.394950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdds9\" (UniqueName: \"kubernetes.io/projected/2dd0c77d-4ae2-41fd-9dd7-176f59aab38e-kube-api-access-gdds9\") pod \"monitoring-plugin-d5dcff7cf-b2xxg\" (UID: \"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.463121 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.463040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:49.598239 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:49.598215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"]
Jul 06 20:50:49.599953 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:50:49.599927    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd0c77d_4ae2_41fd_9dd7_176f59aab38e.slice/crio-cbdd95c6dfb07116ad9f7072511ec9cf54c75c771c62197adb174e6704740a3c WatchSource:0}: Error finding container cbdd95c6dfb07116ad9f7072511ec9cf54c75c771c62197adb174e6704740a3c: Status 404 returned error can't find the container with id cbdd95c6dfb07116ad9f7072511ec9cf54c75c771c62197adb174e6704740a3c
Jul 06 20:50:50.570973 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:50.570931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg" event={"ID":"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e","Type":"ContainerStarted","Data":"cbdd95c6dfb07116ad9f7072511ec9cf54c75c771c62197adb174e6704740a3c"}
Jul 06 20:50:50.688127 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:50.688098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:50.884172 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:50.884087    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:50.996547 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:50.996517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:51.231184 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.231102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:51.426304 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.426266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:51.575377 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.575338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg" event={"ID":"2dd0c77d-4ae2-41fd-9dd7-176f59aab38e","Type":"ContainerStarted","Data":"b9ad17471f408ecc2a7135072be38edf4ae65ce5479664b43da89859432f19c2"}
Jul 06 20:50:51.575797 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.575565    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:51.580321 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.580299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg"
Jul 06 20:50:51.600133 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.600088    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-b2xxg" podStartSLOduration=1.5614084849999998 podStartE2EDuration="2.600073963s" totalImagesPullingTime="1.038665478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:51.597694373 +0000 UTC m=+94.057456249" watchObservedRunningTime="2026-07-06 20:50:51.600073963 +0000 UTC m=+94.059835838"
Jul 06 20:50:51.680135 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.680100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:51.779500 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:51.779471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:52.168064 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:52.168025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:52.243147 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:52.243116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:52.708131 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:52.708098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:52.796021 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:52.795989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:53.328351 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:53.328317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:53.828534 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:53.828500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:53.930125 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:53.930094    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:54.505449 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:54.505424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c7f8c7bcf-nl7sm"
Jul 06 20:50:54.681934 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:54.681900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:54.753483 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:54.753452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:54.935741 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:54.935687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:55.017203 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:55.017161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:57.315605 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:57.315510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:57.379638 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:50:57.379607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:03.501644 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:03.501608    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rnnjh"
Jul 06 20:51:03.920740 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:03.920684    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:03.979794 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:03.979760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:06.941046 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:06.941006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:07.040142 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:07.040111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:08.339679 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:08.339645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:08.439901 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:08.439868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:09.163436 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:09.163402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:09.264526 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:09.264494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:17.701644 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:17.701613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:17.807689 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:17.807658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:17.879034 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:17.879002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:18.666032 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:18.666000    2578 generic.go:356] "Generic (PLEG): container finished" podID="aca76d7d-a60d-4838-98ed-a851fb5b68a8" containerID="615be4b225bfc9b0ca6af198c696288a927035f9bb3b375b3716968fd784c809" exitCode=0
Jul 06 20:51:18.666198 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:18.666040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt" event={"ID":"aca76d7d-a60d-4838-98ed-a851fb5b68a8","Type":"ContainerDied","Data":"615be4b225bfc9b0ca6af198c696288a927035f9bb3b375b3716968fd784c809"}
Jul 06 20:51:18.666400 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:18.666382    2578 scope.go:117] "RemoveContainer" containerID="615be4b225bfc9b0ca6af198c696288a927035f9bb3b375b3716968fd784c809"
Jul 06 20:51:19.671419 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:19.671379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-87rqt" event={"ID":"aca76d7d-a60d-4838-98ed-a851fb5b68a8","Type":"ContainerStarted","Data":"1277555dcfcde9bfef57c0d992d9c05709fe90ba343156f1652317ac51acdfb8"}
Jul 06 20:51:22.719833 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:22.719797    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:22.796548 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:22.796519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:24.522794 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:24.522759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:24.591332 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:24.591298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:24.655171 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:24.655137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:58.333316 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:51:58.333277    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:03.917183 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:03.917150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:04.118958 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:04.118910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:12.844708 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:12.844623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:12.947454 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:12.947418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:13.045776 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:13.045741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:13.574388 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:13.574348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:13.674213 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:13.674182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:24.294288 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:24.294251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:24.458321 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:24.458287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:49.048957 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:49.048921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:49.150093 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:52:49.150060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:19.172301 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:19.172247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:19.239700 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:19.239668    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:31.162219 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:31.162185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:31.240438 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:31.240410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:37.720631 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:37.720592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:37.822198 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:37.822167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:37.919484 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:37.919451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:39.089729 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:39.089689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:39.164828 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:39.164797    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:49.579983 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:49.579949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:49.678178 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:49.678105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:49.758601 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:49.758572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:55.554804 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:55.554771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:55.633044 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:53:55.633016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:17.967747 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:17.967704    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 20:54:24.674329 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:24.674293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:24.770315 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:24.770283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:25.028775 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:25.028747    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:25.125493 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:25.125462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:25.226744 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:25.226696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:47.493786 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:47.493752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:48.578317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:48.578272    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:53.445439 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.445402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:53.740710 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.740605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:53.747622 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.747591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-4dl5x"]
Jul 06 20:54:53.750561 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.750545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.753238 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.753217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 06 20:54:53.753704 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.753688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-swzgz\""
Jul 06 20:54:53.754304 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.754284    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 06 20:54:53.754380 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.754285    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 06 20:54:53.754380 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.754288    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 06 20:54:53.755971 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.755950    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 06 20:54:53.765635 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.765607    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-4dl5x"]
Jul 06 20:54:53.836244 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.836202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x422\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-kube-api-access-8x422\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.836244 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.836248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/44f79a73-2467-4157-a368-9cfd80a846ad-cabundle0\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.836476 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.836312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.937705 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.937663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x422\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-kube-api-access-8x422\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.937705 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.937729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/44f79a73-2467-4157-a368-9cfd80a846ad-cabundle0\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.937943 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.937808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.937943 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:53.937908    2578 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 06 20:54:53.937943 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:53.937926    2578 secret.go:281] references non-existent secret key: ca.crt
Jul 06 20:54:53.937943 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:53.937937    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 06 20:54:53.938092 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:53.937954    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-4dl5x: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 06 20:54:53.938092 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:53.938029    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates podName:44f79a73-2467-4157-a368-9cfd80a846ad nodeName:}" failed. No retries permitted until 2026-07-06 20:54:54.438008858 +0000 UTC m=+336.897770728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates") pod "keda-operator-68bff59c-4dl5x" (UID: "44f79a73-2467-4157-a368-9cfd80a846ad") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 06 20:54:53.938450 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.938428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/44f79a73-2467-4157-a368-9cfd80a846ad-cabundle0\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:53.942464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.942426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:53.947534 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:53.947508    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x422\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-kube-api-access-8x422\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:54.087256 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.087223    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"]
Jul 06 20:54:54.090367 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.090348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.093055 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.093029    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 06 20:54:54.100240 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.100211    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"]
Jul 06 20:54:54.139578 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.139542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntc5\" (UniqueName: \"kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-kube-api-access-nntc5\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.139762 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.139614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-certificates\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.139762 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.139649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.163734 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.163694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:54.240170 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.240131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntc5\" (UniqueName: \"kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-kube-api-access-nntc5\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.240420 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.240395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-certificates\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.240509 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.240478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.240509 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.240488    2578 secret.go:281] references non-existent secret key: tls.crt
Jul 06 20:54:54.240509 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.240504    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 06 20:54:54.240677 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.240522    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv: references non-existent secret key: tls.crt
Jul 06 20:54:54.240677 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.240585    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-certificates podName:ab78ca49-7cbb-4e68-8eff-db5ef83da8f7 nodeName:}" failed. No retries permitted until 2026-07-06 20:54:54.7405681 +0000 UTC m=+337.200329954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-certificates") pod "keda-metrics-apiserver-559f75f947-p4lwv" (UID: "ab78ca49-7cbb-4e68-8eff-db5ef83da8f7") : references non-existent secret key: tls.crt
Jul 06 20:54:54.240930 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.240911    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.248995 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.248964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntc5\" (UniqueName: \"kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-kube-api-access-nntc5\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.265572 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.265543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:54.442428 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.442334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:54.442584 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.442482    2578 secret.go:281] references non-existent secret key: ca.crt
Jul 06 20:54:54.442584 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.442500    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 06 20:54:54.442584 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.442509    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-4dl5x: references non-existent secret key: ca.crt
Jul 06 20:54:54.442584 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:54:54.442557    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates podName:44f79a73-2467-4157-a368-9cfd80a846ad nodeName:}" failed. No retries permitted until 2026-07-06 20:54:55.442544127 +0000 UTC m=+337.902305981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates") pod "keda-operator-68bff59c-4dl5x" (UID: "44f79a73-2467-4157-a368-9cfd80a846ad") : references non-existent secret key: ca.crt
Jul 06 20:54:54.613920 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.613879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:54.711812 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.711707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:54.745005 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.744967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-certificates\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.747642 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.747612    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ab78ca49-7cbb-4e68-8eff-db5ef83da8f7-certificates\") pod \"keda-metrics-apiserver-559f75f947-p4lwv\" (UID: \"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:54.827176 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:54.827143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:55.002173 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.002080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:54:55.014117 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.014079    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:55.137889 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.137860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"]
Jul 06 20:54:55.140789 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:54:55.140751    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab78ca49_7cbb_4e68_8eff_db5ef83da8f7.slice/crio-551e93c0597a1f22592d486c6655c4534fe862384c9758dc47d18e0e60aa6848 WatchSource:0}: Error finding container 551e93c0597a1f22592d486c6655c4534fe862384c9758dc47d18e0e60aa6848: Status 404 returned error can't find the container with id 551e93c0597a1f22592d486c6655c4534fe862384c9758dc47d18e0e60aa6848
Jul 06 20:54:55.142031 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.142012    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:54:55.215164 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.215123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:55.313435 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.313400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:55.379919 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.379883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv" event={"ID":"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7","Type":"ContainerStarted","Data":"551e93c0597a1f22592d486c6655c4534fe862384c9758dc47d18e0e60aa6848"}
Jul 06 20:54:55.414363 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.414337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:55.450021 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.449984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:55.452397 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.452375    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/44f79a73-2467-4157-a368-9cfd80a846ad-certificates\") pod \"keda-operator-68bff59c-4dl5x\" (UID: \"44f79a73-2467-4157-a368-9cfd80a846ad\") " pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:55.560639 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.560594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:54:55.691317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:55.691261    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-4dl5x"]
Jul 06 20:54:55.693964 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:54:55.693942    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f79a73_2467_4157_a368_9cfd80a846ad.slice/crio-1a1db5d037cc2ca14c8793941f0403c1752f29d85c47644ba96da1a0c49e9d49 WatchSource:0}: Error finding container 1a1db5d037cc2ca14c8793941f0403c1752f29d85c47644ba96da1a0c49e9d49: Status 404 returned error can't find the container with id 1a1db5d037cc2ca14c8793941f0403c1752f29d85c47644ba96da1a0c49e9d49
Jul 06 20:54:56.385270 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:54:56.385225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-4dl5x" event={"ID":"44f79a73-2467-4157-a368-9cfd80a846ad","Type":"ContainerStarted","Data":"1a1db5d037cc2ca14c8793941f0403c1752f29d85c47644ba96da1a0c49e9d49"}
Jul 06 20:55:00.406462 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:00.406409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-4dl5x" event={"ID":"44f79a73-2467-4157-a368-9cfd80a846ad","Type":"ContainerStarted","Data":"a9e3278e786f0431bc3229bdef05de6a27912ccb34a09fe3c702d34bef7e6215"}
Jul 06 20:55:00.406983 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:00.406525    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:55:00.407986 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:00.407964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv" event={"ID":"ab78ca49-7cbb-4e68-8eff-db5ef83da8f7","Type":"ContainerStarted","Data":"2ec846e52319fcb42d14f4a6c5ed6bf59e3582487cf070772aa261088f05fb6a"}
Jul 06 20:55:00.408126 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:00.408022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:55:00.425485 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:00.425437    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-4dl5x" podStartSLOduration=3.111755352 podStartE2EDuration="7.425420563s" totalImagesPullingTime="4.313665211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:55:00.424102903 +0000 UTC m=+342.883864780" watchObservedRunningTime="2026-07-06 20:55:00.425420563 +0000 UTC m=+342.885182441"
Jul 06 20:55:00.440169 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:00.440113    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv" podStartSLOduration=1.578495237 podStartE2EDuration="6.440096837s" totalImagesPullingTime="4.8616016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:55:00.43959543 +0000 UTC m=+342.899357329" watchObservedRunningTime="2026-07-06 20:55:00.440096837 +0000 UTC m=+342.899858712"
Jul 06 20:55:06.993169 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:06.993113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:11.416026 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:11.415948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p4lwv"
Jul 06 20:55:16.001767 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:16.001730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:21.414299 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:21.414265    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-4dl5x"
Jul 06 20:55:59.368076 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:55:59.368038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:05.823287 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.823245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:05.928909 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.928880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:05.985952 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.985919    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-l9wwh"]
Jul 06 20:56:05.988200 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.988183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:05.991381 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.991354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 06 20:56:05.992611 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.992586    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 06 20:56:05.993480 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.993464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hzhjx\""
Jul 06 20:56:05.994046 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:05.994025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 06 20:56:06.002613 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.002589    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-l9wwh"]
Jul 06 20:56:06.006460 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.006434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:06.074165 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.074077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:06.092223 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.092191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/92b6ed37-2176-4efb-9e65-a3b455dba9c5-data\") pod \"seaweedfs-64568bcd49-l9wwh\" (UID: \"92b6ed37-2176-4efb-9e65-a3b455dba9c5\") " pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:06.092349 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.092266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g78xm\" (UniqueName: \"kubernetes.io/projected/92b6ed37-2176-4efb-9e65-a3b455dba9c5-kube-api-access-g78xm\") pod \"seaweedfs-64568bcd49-l9wwh\" (UID: \"92b6ed37-2176-4efb-9e65-a3b455dba9c5\") " pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:06.193488 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.193451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/92b6ed37-2176-4efb-9e65-a3b455dba9c5-data\") pod \"seaweedfs-64568bcd49-l9wwh\" (UID: \"92b6ed37-2176-4efb-9e65-a3b455dba9c5\") " pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:06.193667 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.193504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g78xm\" (UniqueName: \"kubernetes.io/projected/92b6ed37-2176-4efb-9e65-a3b455dba9c5-kube-api-access-g78xm\") pod \"seaweedfs-64568bcd49-l9wwh\" (UID: \"92b6ed37-2176-4efb-9e65-a3b455dba9c5\") " pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:06.193964 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.193942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/92b6ed37-2176-4efb-9e65-a3b455dba9c5-data\") pod \"seaweedfs-64568bcd49-l9wwh\" (UID: \"92b6ed37-2176-4efb-9e65-a3b455dba9c5\") " pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:06.202637 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.202608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g78xm\" (UniqueName: \"kubernetes.io/projected/92b6ed37-2176-4efb-9e65-a3b455dba9c5-kube-api-access-g78xm\") pod \"seaweedfs-64568bcd49-l9wwh\" (UID: \"92b6ed37-2176-4efb-9e65-a3b455dba9c5\") " pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:06.299754 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.299704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:06.392539 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.392505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:06.437226 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.437175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-l9wwh"]
Jul 06 20:56:06.442248 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:56:06.442214    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b6ed37_2176_4efb_9e65_a3b455dba9c5.slice/crio-da398027a90b49af6e142416a65a8e86795bea5766e4151d78849b21fd4c7079 WatchSource:0}: Error finding container da398027a90b49af6e142416a65a8e86795bea5766e4151d78849b21fd4c7079: Status 404 returned error can't find the container with id da398027a90b49af6e142416a65a8e86795bea5766e4151d78849b21fd4c7079
Jul 06 20:56:06.519253 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.519221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:06.616613 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.616527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:06.627733 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.627696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-l9wwh" event={"ID":"92b6ed37-2176-4efb-9e65-a3b455dba9c5","Type":"ContainerStarted","Data":"da398027a90b49af6e142416a65a8e86795bea5766e4151d78849b21fd4c7079"}
Jul 06 20:56:06.676988 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:06.676954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:10.643248 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:10.643213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-l9wwh" event={"ID":"92b6ed37-2176-4efb-9e65-a3b455dba9c5","Type":"ContainerStarted","Data":"f9e3249ff9f18ce8c5e0a1690792e5093e750a2861b48f037fc585f1b0b3ba40"}
Jul 06 20:56:10.643634 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:10.643355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:10.661618 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:10.661568    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-l9wwh" podStartSLOduration=2.445147557 podStartE2EDuration="5.661552641s" totalImagesPullingTime="3.216405084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:56:10.661121514 +0000 UTC m=+413.120883422" watchObservedRunningTime="2026-07-06 20:56:10.661552641 +0000 UTC m=+413.121314533"
Jul 06 20:56:16.648914 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:16.648877    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-l9wwh"
Jul 06 20:56:39.899986 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:39.899907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:39.971531 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:39.971495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:41.844651 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:41.844622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:41.954367 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:41.954332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:43.562159 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:43.562126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:43.663255 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:56:43.663226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:00.007865 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:00.007831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:01.278332 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:01.278299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:02.015150 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.015119    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"]
Jul 06 20:57:02.019058 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.019039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.021763 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.021739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vxn92\""
Jul 06 20:57:02.022369 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.022349    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 06 20:57:02.022869 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.022854    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:57:02.034004 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.033977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"]
Jul 06 20:57:02.117710 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.117665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b31eab07-fa57-45ed-beef-cfc1f2303ddb-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7r4wl\" (UID: \"b31eab07-fa57-45ed-beef-cfc1f2303ddb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.117710 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.117713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56jw\" (UniqueName: \"kubernetes.io/projected/b31eab07-fa57-45ed-beef-cfc1f2303ddb-kube-api-access-b56jw\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7r4wl\" (UID: \"b31eab07-fa57-45ed-beef-cfc1f2303ddb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.218529 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.218491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b31eab07-fa57-45ed-beef-cfc1f2303ddb-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7r4wl\" (UID: \"b31eab07-fa57-45ed-beef-cfc1f2303ddb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.218529 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.218529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56jw\" (UniqueName: \"kubernetes.io/projected/b31eab07-fa57-45ed-beef-cfc1f2303ddb-kube-api-access-b56jw\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7r4wl\" (UID: \"b31eab07-fa57-45ed-beef-cfc1f2303ddb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.218969 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.218947    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b31eab07-fa57-45ed-beef-cfc1f2303ddb-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7r4wl\" (UID: \"b31eab07-fa57-45ed-beef-cfc1f2303ddb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.227417 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.227394    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56jw\" (UniqueName: \"kubernetes.io/projected/b31eab07-fa57-45ed-beef-cfc1f2303ddb-kube-api-access-b56jw\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7r4wl\" (UID: \"b31eab07-fa57-45ed-beef-cfc1f2303ddb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.327854 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.327817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"
Jul 06 20:57:02.463940 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.463888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl"]
Jul 06 20:57:02.467046 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:57:02.467019    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31eab07_fa57_45ed_beef_cfc1f2303ddb.slice/crio-9d00601b10ca9dfea0872248d1e141bbfdf98359f198cd25b49c90bbc8cbaae1 WatchSource:0}: Error finding container 9d00601b10ca9dfea0872248d1e141bbfdf98359f198cd25b49c90bbc8cbaae1: Status 404 returned error can't find the container with id 9d00601b10ca9dfea0872248d1e141bbfdf98359f198cd25b49c90bbc8cbaae1
Jul 06 20:57:02.820329 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:02.820297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl" event={"ID":"b31eab07-fa57-45ed-beef-cfc1f2303ddb","Type":"ContainerStarted","Data":"9d00601b10ca9dfea0872248d1e141bbfdf98359f198cd25b49c90bbc8cbaae1"}
Jul 06 20:57:03.614208 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:03.614154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:03.714845 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:03.714809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:04.357110 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:04.357074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:04.451185 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:04.451142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:05.796242 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:05.796205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:05.834464 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:05.834430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl" event={"ID":"b31eab07-fa57-45ed-beef-cfc1f2303ddb","Type":"ContainerStarted","Data":"6811ee103a2d044e85ac0ecaae9a41357d4ed5c85473534c97b76bdaf25f42cb"}
Jul 06 20:57:05.894893 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:05.894857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:08.410730 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:08.410680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:08.510905 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:08.510873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:08.610549 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:08.610510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:10.258234 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:10.258203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:10.363753 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:10.362037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:10.454687 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:10.454655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:13.978786 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:13.978709    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7r4wl" podStartSLOduration=10.376181462 podStartE2EDuration="12.978690801s" totalImagesPullingTime="2.602509339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:05.889472801 +0000 UTC m=+468.349234678" watchObservedRunningTime="2026-07-06 20:57:13.978690801 +0000 UTC m=+476.438452673"
Jul 06 20:57:13.979420 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:13.979403    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k5588"]
Jul 06 20:57:13.982409 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:13.982390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:13.985174 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:13.985156    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 06 20:57:13.986362 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:13.986344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2j5cw\""
Jul 06 20:57:13.986420 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:13.986406    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 06 20:57:13.992637 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:13.992610    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k5588"]
Jul 06 20:57:14.119535 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.119492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6f68904-e924-4767-9502-9f76d741cd21-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k5588\" (UID: \"e6f68904-e924-4767-9502-9f76d741cd21\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:14.119744 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.119607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jrd\" (UniqueName: \"kubernetes.io/projected/e6f68904-e924-4767-9502-9f76d741cd21-kube-api-access-45jrd\") pod \"cert-manager-webhook-6486ffd997-k5588\" (UID: \"e6f68904-e924-4767-9502-9f76d741cd21\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:14.220906 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.220862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jrd\" (UniqueName: \"kubernetes.io/projected/e6f68904-e924-4767-9502-9f76d741cd21-kube-api-access-45jrd\") pod \"cert-manager-webhook-6486ffd997-k5588\" (UID: \"e6f68904-e924-4767-9502-9f76d741cd21\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:14.221103 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.220931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6f68904-e924-4767-9502-9f76d741cd21-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k5588\" (UID: \"e6f68904-e924-4767-9502-9f76d741cd21\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:14.232678 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.232608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6f68904-e924-4767-9502-9f76d741cd21-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k5588\" (UID: \"e6f68904-e924-4767-9502-9f76d741cd21\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:14.232824 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.232806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jrd\" (UniqueName: \"kubernetes.io/projected/e6f68904-e924-4767-9502-9f76d741cd21-kube-api-access-45jrd\") pod \"cert-manager-webhook-6486ffd997-k5588\" (UID: \"e6f68904-e924-4767-9502-9f76d741cd21\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:14.291570 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.291534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:14.426701 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.426667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k5588"]
Jul 06 20:57:14.429389 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:57:14.429363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6f68904_e924_4767_9502_9f76d741cd21.slice/crio-31b5787a2d256446587e841d95b57913cdb476883421350e1e93292936dcc08f WatchSource:0}: Error finding container 31b5787a2d256446587e841d95b57913cdb476883421350e1e93292936dcc08f: Status 404 returned error can't find the container with id 31b5787a2d256446587e841d95b57913cdb476883421350e1e93292936dcc08f
Jul 06 20:57:14.869887 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:14.869852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-k5588" event={"ID":"e6f68904-e924-4767-9502-9f76d741cd21","Type":"ContainerStarted","Data":"31b5787a2d256446587e841d95b57913cdb476883421350e1e93292936dcc08f"}
Jul 06 20:57:15.609435 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:15.609403    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:17.274326 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:17.274296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:17.332187 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:17.332140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:17.884702 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:17.884669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-k5588" event={"ID":"e6f68904-e924-4767-9502-9f76d741cd21","Type":"ContainerStarted","Data":"f8ddda561facab18475dc67e079d34b6cbb5c7eba1659cef25228dc19c76e533"}
Jul 06 20:57:17.884898 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:17.884736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:17.904805 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:17.904753    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-k5588" podStartSLOduration=2.319087746 podStartE2EDuration="4.904734175s" totalImagesPullingTime="2.585646429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:17.903541036 +0000 UTC m=+480.363302912" watchObservedRunningTime="2026-07-06 20:57:17.904734175 +0000 UTC m=+480.364496048"
Jul 06 20:57:19.688506 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:19.688469    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:19.819327 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:19.819290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:19.880742 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:19.880691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:20.423936 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:20.423901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:21.301183 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:21.301137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:21.402813 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:21.402776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:23.150859 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.150825    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"]
Jul 06 20:57:23.154165 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.154144    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.157797 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.157773    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wc8n5\""
Jul 06 20:57:23.157945 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.157800    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:57:23.157945 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.157895    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 20:57:23.163638 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.163618    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"]
Jul 06 20:57:23.196534 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.196490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc75e51e-134d-4d69-acde-b2f17af5fabb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7nlh5\" (UID: \"cc75e51e-134d-4d69-acde-b2f17af5fabb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.196534 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.196533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4wn\" (UniqueName: \"kubernetes.io/projected/cc75e51e-134d-4d69-acde-b2f17af5fabb-kube-api-access-gk4wn\") pod \"openshift-lws-operator-fd8ccff4c-7nlh5\" (UID: \"cc75e51e-134d-4d69-acde-b2f17af5fabb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.297133 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.297095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc75e51e-134d-4d69-acde-b2f17af5fabb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7nlh5\" (UID: \"cc75e51e-134d-4d69-acde-b2f17af5fabb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.297133 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.297137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4wn\" (UniqueName: \"kubernetes.io/projected/cc75e51e-134d-4d69-acde-b2f17af5fabb-kube-api-access-gk4wn\") pod \"openshift-lws-operator-fd8ccff4c-7nlh5\" (UID: \"cc75e51e-134d-4d69-acde-b2f17af5fabb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.297525 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.297504    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc75e51e-134d-4d69-acde-b2f17af5fabb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7nlh5\" (UID: \"cc75e51e-134d-4d69-acde-b2f17af5fabb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.310204 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.310173    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4wn\" (UniqueName: \"kubernetes.io/projected/cc75e51e-134d-4d69-acde-b2f17af5fabb-kube-api-access-gk4wn\") pod \"openshift-lws-operator-fd8ccff4c-7nlh5\" (UID: \"cc75e51e-134d-4d69-acde-b2f17af5fabb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.463301 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.463195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"
Jul 06 20:57:23.598779 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.598744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5"]
Jul 06 20:57:23.602239 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:57:23.602211    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc75e51e_134d_4d69_acde_b2f17af5fabb.slice/crio-df30f293dbd4f56928ebb1e67442372f3d5d4d84f43c1e9dfaffa59dceeb608f WatchSource:0}: Error finding container df30f293dbd4f56928ebb1e67442372f3d5d4d84f43c1e9dfaffa59dceeb608f: Status 404 returned error can't find the container with id df30f293dbd4f56928ebb1e67442372f3d5d4d84f43c1e9dfaffa59dceeb608f
Jul 06 20:57:23.891020 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.890985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-k5588"
Jul 06 20:57:23.908357 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:23.908320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5" event={"ID":"cc75e51e-134d-4d69-acde-b2f17af5fabb","Type":"ContainerStarted","Data":"df30f293dbd4f56928ebb1e67442372f3d5d4d84f43c1e9dfaffa59dceeb608f"}
Jul 06 20:57:26.922448 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:26.922357    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5" event={"ID":"cc75e51e-134d-4d69-acde-b2f17af5fabb","Type":"ContainerStarted","Data":"9cff0e0331805f4c21ffef2f8b8386829d28435d4e9d6fbc5fe091a239f7166f"}
Jul 06 20:57:26.940364 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:26.940312    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7nlh5" podStartSLOduration=0.991203956 podStartE2EDuration="3.940294083s" totalImagesPullingTime="2.949090127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:26.938146672 +0000 UTC m=+489.397908560" watchObservedRunningTime="2026-07-06 20:57:26.940294083 +0000 UTC m=+489.400055960"
Jul 06 20:57:30.463685 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:30.463654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:30.571036 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:30.571001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:30.630254 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:30.630221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:33.789945 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:33.789880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:39.120351 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:39.120317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:39.197028 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:39.197000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:40.337603 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:40.337570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:42.817467 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:42.817430    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:42.919240 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:42.919201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:46.873240 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.873207    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"]
Jul 06 20:57:46.876487 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.876462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.879202 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.879153    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 20:57:46.879202 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.879152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 20:57:46.879413 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.879310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 20:57:46.879491 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.879441    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-2ddf6\""
Jul 06 20:57:46.890043 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.890016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"]
Jul 06 20:57:46.919626 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.919588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:46.949969 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.949928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.949969 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.949976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07976c37-1aab-44af-ab95-9824a1a523b4-istio-token\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.950229 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.950013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07976c37-1aab-44af-ab95-9824a1a523b4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.950229 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.950040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07976c37-1aab-44af-ab95-9824a1a523b4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.950229 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.950065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-workload-socket\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.950229 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.950102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-credential-socket\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.950229 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.950132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-workload-certs\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.950229 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.950190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82q8x\" (UniqueName: \"kubernetes.io/projected/07976c37-1aab-44af-ab95-9824a1a523b4-kube-api-access-82q8x\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:46.950229 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:46.950220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-istio-data\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.014067 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.014033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:47.051359 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-workload-certs\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051514 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82q8x\" (UniqueName: \"kubernetes.io/projected/07976c37-1aab-44af-ab95-9824a1a523b4-kube-api-access-82q8x\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051514 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-istio-data\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051514 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051514 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07976c37-1aab-44af-ab95-9824a1a523b4-istio-token\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051514 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07976c37-1aab-44af-ab95-9824a1a523b4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051786 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07976c37-1aab-44af-ab95-9824a1a523b4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051786 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-workload-socket\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051786 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-credential-socket\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051948 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051829    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-workload-certs\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.051948 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051906    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-istio-data\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.052065 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.051963    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-credential-socket\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.052125 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.052099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-workload-socket\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.052284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.052261    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07976c37-1aab-44af-ab95-9824a1a523b4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.054067 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.054036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07976c37-1aab-44af-ab95-9824a1a523b4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.054239 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.054220    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07976c37-1aab-44af-ab95-9824a1a523b4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.064875 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.064848    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07976c37-1aab-44af-ab95-9824a1a523b4-istio-token\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.067596 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.067573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82q8x\" (UniqueName: \"kubernetes.io/projected/07976c37-1aab-44af-ab95-9824a1a523b4-kube-api-access-82q8x\") pod \"openshift-ai-inference-openshift-default-769cdd8d88-hxzrg\" (UID: \"07976c37-1aab-44af-ab95-9824a1a523b4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.187531 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.187428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:47.340042 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:47.339978    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"]
Jul 06 20:57:47.342004 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:57:47.341984    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07976c37_1aab_44af_ab95_9824a1a523b4.slice/crio-1f6778496acad76f2887a44c5bf101d7deb4edf3ae6bf1ca413e7b1fcf06bd3b WatchSource:0}: Error finding container 1f6778496acad76f2887a44c5bf101d7deb4edf3ae6bf1ca413e7b1fcf06bd3b: Status 404 returned error can't find the container with id 1f6778496acad76f2887a44c5bf101d7deb4edf3ae6bf1ca413e7b1fcf06bd3b
Jul 06 20:57:48.003224 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:48.003188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" event={"ID":"07976c37-1aab-44af-ab95-9824a1a523b4","Type":"ContainerStarted","Data":"1f6778496acad76f2887a44c5bf101d7deb4edf3ae6bf1ca413e7b1fcf06bd3b"}
Jul 06 20:57:49.990874 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:49.990830    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 20:57:49.991266 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:49.990914    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 20:57:49.991266 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:49.990970    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 20:57:50.041137 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.041110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:50.111667 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.111633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:50.123532 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.123486    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"]
Jul 06 20:57:50.127063 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.127037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.129939 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.129916    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 06 20:57:50.129939 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.129934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 06 20:57:50.131060 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.131040    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 06 20:57:50.131165 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.131099    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b66t2\""
Jul 06 20:57:50.136794 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.136771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"]
Jul 06 20:57:50.177601 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.177569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40703173-9342-41ca-ba6c-95aab8dc88d6-manager-config\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.177801 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.177609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40703173-9342-41ca-ba6c-95aab8dc88d6-cert\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.177801 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.177668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqxrx\" (UniqueName: \"kubernetes.io/projected/40703173-9342-41ca-ba6c-95aab8dc88d6-kube-api-access-fqxrx\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.177801 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.177694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40703173-9342-41ca-ba6c-95aab8dc88d6-metrics-cert\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.279044 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.279007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40703173-9342-41ca-ba6c-95aab8dc88d6-cert\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.279213 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.279067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqxrx\" (UniqueName: \"kubernetes.io/projected/40703173-9342-41ca-ba6c-95aab8dc88d6-kube-api-access-fqxrx\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.279213 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.279102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40703173-9342-41ca-ba6c-95aab8dc88d6-metrics-cert\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.279213 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.279136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40703173-9342-41ca-ba6c-95aab8dc88d6-manager-config\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.279894 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.279873    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40703173-9342-41ca-ba6c-95aab8dc88d6-manager-config\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.281608 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.281576    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40703173-9342-41ca-ba6c-95aab8dc88d6-metrics-cert\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.281707 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.281605    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40703173-9342-41ca-ba6c-95aab8dc88d6-cert\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.289181 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.289154    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqxrx\" (UniqueName: \"kubernetes.io/projected/40703173-9342-41ca-ba6c-95aab8dc88d6-kube-api-access-fqxrx\") pod \"lws-controller-manager-7694784d57-vvk2h\" (UID: \"40703173-9342-41ca-ba6c-95aab8dc88d6\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.439234 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.439193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:50.574925 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.574902    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"]
Jul 06 20:57:50.577615 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:57:50.577584    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40703173_9342_41ca_ba6c_95aab8dc88d6.slice/crio-f2478849899fd9de93619eaf007ba81553c315de69b295e17727e5cf78959b87 WatchSource:0}: Error finding container f2478849899fd9de93619eaf007ba81553c315de69b295e17727e5cf78959b87: Status 404 returned error can't find the container with id f2478849899fd9de93619eaf007ba81553c315de69b295e17727e5cf78959b87
Jul 06 20:57:50.610972 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:50.610936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:51.018915 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:51.018822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h" event={"ID":"40703173-9342-41ca-ba6c-95aab8dc88d6","Type":"ContainerStarted","Data":"f2478849899fd9de93619eaf007ba81553c315de69b295e17727e5cf78959b87"}
Jul 06 20:57:51.020221 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:51.020194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" event={"ID":"07976c37-1aab-44af-ab95-9824a1a523b4","Type":"ContainerStarted","Data":"2bf489327374fe1de0487cf78a460fb52a00218459251dcb4eb06de8b3cb2a47"}
Jul 06 20:57:51.044015 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:51.043952    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podStartSLOduration=2.397506553 podStartE2EDuration="5.043934348s" totalImagesPullingTime="2.646427795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:51.043492092 +0000 UTC m=+513.503253968" watchObservedRunningTime="2026-07-06 20:57:51.043934348 +0000 UTC m=+513.503696224"
Jul 06 20:57:51.188251 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:51.188214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:51.193191 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:51.193163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:52.025129 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:52.025085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:52.026243 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:52.026217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 20:57:52.592405 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:52.592365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:52.697659 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:52.697622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:53.030682 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:53.030631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h" event={"ID":"40703173-9342-41ca-ba6c-95aab8dc88d6","Type":"ContainerStarted","Data":"1a9eda3aaa79cad7df8b683c23f82c496809d47e33d8a5c107c38ae4c9ada9d7"}
Jul 06 20:57:53.031089 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:53.030879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:57:53.050170 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:57:53.050122    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h" podStartSLOduration=1.47390682 podStartE2EDuration="3.050105772s" totalImagesPullingTime="1.576198952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:53.049008928 +0000 UTC m=+515.508770805" watchObservedRunningTime="2026-07-06 20:57:53.050105772 +0000 UTC m=+515.509867647"
Jul 06 20:58:04.037806 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:04.037775    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7694784d57-vvk2h"
Jul 06 20:58:04.232361 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:04.232327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:06.646516 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:06.646476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:07.032729 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:07.032683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:08.077324 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:08.077292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:08.183232 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:08.183198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:09.358450 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:09.358360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:09.848191 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:09.848159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:10.245335 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:10.245249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:10.701863 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:10.701829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:10.994034 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:10.993946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:11.108351 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:11.108320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:14.258903 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:14.258873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:15.322915 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:15.322886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:17.353881 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:17.353849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:17.461327 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:17.461293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:20.834059 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:20.834027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:20.932706 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:20.932674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:22.773433 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:22.773395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:22.874967 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:22.874933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:22.978162 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:22.978132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:23.238388 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:23.238305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:23.341816 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:23.341781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:26.784150 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:26.784110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:26.884322 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:26.884290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:26.984846 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:26.984810    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:33.904051 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:33.904017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:34.003222 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:34.003187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:36.520512 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:36.520475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:36.619997 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:36.619964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:38.824640 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:38.824604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:38.936093 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:38.936058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:39.031081 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:39.031043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:41.896371 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:41.896333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:41.996556 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:41.996525    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:42.300305 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:42.300269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:42.392474 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:42.392438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:46.653614 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:46.653577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:46.749772 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:46.749735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:46.850674 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:46.850644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:53.730192 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:53.730154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:53.833110 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:58:53.833080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:05.594575 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:05.594542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:05.726239 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:05.726205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:05.838159 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:05.838125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:06.610013 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.609969    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kw7ft"]
Jul 06 20:59:06.612223 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.612200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kw7ft"
Jul 06 20:59:06.616215 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.616191    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 20:59:06.617141 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.617124    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4n9bd\""
Jul 06 20:59:06.617284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.617268    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 20:59:06.629780 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.629739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kw7ft"]
Jul 06 20:59:06.652284 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.652251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:06.716828 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.716784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c5j\" (UniqueName: \"kubernetes.io/projected/2331da34-314f-4713-be9b-7d93e273feb2-kube-api-access-r5c5j\") pod \"authorino-b6c89bb85-kw7ft\" (UID: \"2331da34-314f-4713-be9b-7d93e273feb2\") " pod="kuadrant-system/authorino-b6c89bb85-kw7ft"
Jul 06 20:59:06.723838 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.723811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:06.818230 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.818199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c5j\" (UniqueName: \"kubernetes.io/projected/2331da34-314f-4713-be9b-7d93e273feb2-kube-api-access-r5c5j\") pod \"authorino-b6c89bb85-kw7ft\" (UID: \"2331da34-314f-4713-be9b-7d93e273feb2\") " pod="kuadrant-system/authorino-b6c89bb85-kw7ft"
Jul 06 20:59:06.830374 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.830339    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c5j\" (UniqueName: \"kubernetes.io/projected/2331da34-314f-4713-be9b-7d93e273feb2-kube-api-access-r5c5j\") pod \"authorino-b6c89bb85-kw7ft\" (UID: \"2331da34-314f-4713-be9b-7d93e273feb2\") " pod="kuadrant-system/authorino-b6c89bb85-kw7ft"
Jul 06 20:59:06.921015 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:06.920930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kw7ft"
Jul 06 20:59:07.066837 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:07.066811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kw7ft"]
Jul 06 20:59:07.069070 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:59:07.069045    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2331da34_314f_4713_be9b_7d93e273feb2.slice/crio-92cffa4f0446f49345e30952797772e2586b611bf3da7ea95cee21238deb16f6 WatchSource:0}: Error finding container 92cffa4f0446f49345e30952797772e2586b611bf3da7ea95cee21238deb16f6: Status 404 returned error can't find the container with id 92cffa4f0446f49345e30952797772e2586b611bf3da7ea95cee21238deb16f6
Jul 06 20:59:07.327188 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:07.327147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kw7ft" event={"ID":"2331da34-314f-4713-be9b-7d93e273feb2","Type":"ContainerStarted","Data":"92cffa4f0446f49345e30952797772e2586b611bf3da7ea95cee21238deb16f6"}
Jul 06 20:59:10.342395 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:10.342358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kw7ft" event={"ID":"2331da34-314f-4713-be9b-7d93e273feb2","Type":"ContainerStarted","Data":"45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2"}
Jul 06 20:59:10.365124 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:10.365068    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-kw7ft" podStartSLOduration=2.080805621 podStartE2EDuration="4.365051322s" totalImagesPullingTime="2.284245701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:10.36449428 +0000 UTC m=+592.824256157" watchObservedRunningTime="2026-07-06 20:59:10.365051322 +0000 UTC m=+592.824813197"
Jul 06 20:59:12.363439 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:12.363401    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kw7ft"]
Jul 06 20:59:12.363861 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:12.363555    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-kw7ft" podUID="2331da34-314f-4713-be9b-7d93e273feb2" containerName="authorino" containerID="cri-o://45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2" gracePeriod=30
Jul 06 20:59:12.609397 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:12.609368    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kw7ft"
Jul 06 20:59:12.771928 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:12.771890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5c5j\" (UniqueName: \"kubernetes.io/projected/2331da34-314f-4713-be9b-7d93e273feb2-kube-api-access-r5c5j\") pod \"2331da34-314f-4713-be9b-7d93e273feb2\" (UID: \"2331da34-314f-4713-be9b-7d93e273feb2\") "
Jul 06 20:59:12.774100 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:12.774068    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2331da34-314f-4713-be9b-7d93e273feb2-kube-api-access-r5c5j" (OuterVolumeSpecName: "kube-api-access-r5c5j") pod "2331da34-314f-4713-be9b-7d93e273feb2" (UID: "2331da34-314f-4713-be9b-7d93e273feb2"). InnerVolumeSpecName "kube-api-access-r5c5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:59:12.873439 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:12.873402    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c5j\" (UniqueName: \"kubernetes.io/projected/2331da34-314f-4713-be9b-7d93e273feb2-kube-api-access-r5c5j\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 20:59:13.355893 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.355855    2578 generic.go:356] "Generic (PLEG): container finished" podID="2331da34-314f-4713-be9b-7d93e273feb2" containerID="45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2" exitCode=0
Jul 06 20:59:13.356098 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.355902    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kw7ft"
Jul 06 20:59:13.356098 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.355940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kw7ft" event={"ID":"2331da34-314f-4713-be9b-7d93e273feb2","Type":"ContainerDied","Data":"45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2"}
Jul 06 20:59:13.356098 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.355994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kw7ft" event={"ID":"2331da34-314f-4713-be9b-7d93e273feb2","Type":"ContainerDied","Data":"92cffa4f0446f49345e30952797772e2586b611bf3da7ea95cee21238deb16f6"}
Jul 06 20:59:13.356098 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.356018    2578 scope.go:117] "RemoveContainer" containerID="45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2"
Jul 06 20:59:13.364812 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.364758    2578 scope.go:117] "RemoveContainer" containerID="45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2"
Jul 06 20:59:13.365067 ip-10-0-140-38 kubenswrapper[2578]: E0706 20:59:13.365047    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2\": container with ID starting with 45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2 not found: ID does not exist" containerID="45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2"
Jul 06 20:59:13.365107 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.365077    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2"} err="failed to get container status \"45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2\": rpc error: code = NotFound desc = could not find container \"45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2\": container with ID starting with 45c70cbb83b3efe0759821ec533d51123d65239e34c2c3f5c71c950575a262c2 not found: ID does not exist"
Jul 06 20:59:13.383076 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.383047    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kw7ft"]
Jul 06 20:59:13.388901 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:13.388872    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kw7ft"]
Jul 06 20:59:14.159827 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:14.159796    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2331da34-314f-4713-be9b-7d93e273feb2" path="/var/lib/kubelet/pods/2331da34-314f-4713-be9b-7d93e273feb2/volumes"
Jul 06 20:59:19.383350 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:19.383304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:19.472337 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:19.472303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:29.527053 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:29.527018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:29.744320 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:29.744285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:34.247310 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.247272    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-k7wx6"]
Jul 06 20:59:34.247707 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.247580    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2331da34-314f-4713-be9b-7d93e273feb2" containerName="authorino"
Jul 06 20:59:34.247707 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.247594    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2331da34-314f-4713-be9b-7d93e273feb2" containerName="authorino"
Jul 06 20:59:34.247707 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.247653    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2331da34-314f-4713-be9b-7d93e273feb2" containerName="authorino"
Jul 06 20:59:34.250444 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.250426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.253187 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.253160    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 20:59:34.254428 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.254406    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 20:59:34.254563 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.254411    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 06 20:59:34.254563 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.254447    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4n9bd\""
Jul 06 20:59:34.260538 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.260515    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-k7wx6"]
Jul 06 20:59:34.358536 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.358489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b83dded-aa86-4220-939a-e62e9b913f1e-tls-cert\") pod \"authorino-7c77957575-k7wx6\" (UID: \"0b83dded-aa86-4220-939a-e62e9b913f1e\") " pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.358536 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.358546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljj6\" (UniqueName: \"kubernetes.io/projected/0b83dded-aa86-4220-939a-e62e9b913f1e-kube-api-access-mljj6\") pod \"authorino-7c77957575-k7wx6\" (UID: \"0b83dded-aa86-4220-939a-e62e9b913f1e\") " pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.459317 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.459270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b83dded-aa86-4220-939a-e62e9b913f1e-tls-cert\") pod \"authorino-7c77957575-k7wx6\" (UID: \"0b83dded-aa86-4220-939a-e62e9b913f1e\") " pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.459521 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.459338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljj6\" (UniqueName: \"kubernetes.io/projected/0b83dded-aa86-4220-939a-e62e9b913f1e-kube-api-access-mljj6\") pod \"authorino-7c77957575-k7wx6\" (UID: \"0b83dded-aa86-4220-939a-e62e9b913f1e\") " pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.461701 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.461681    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b83dded-aa86-4220-939a-e62e9b913f1e-tls-cert\") pod \"authorino-7c77957575-k7wx6\" (UID: \"0b83dded-aa86-4220-939a-e62e9b913f1e\") " pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.468802 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.468777    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljj6\" (UniqueName: \"kubernetes.io/projected/0b83dded-aa86-4220-939a-e62e9b913f1e-kube-api-access-mljj6\") pod \"authorino-7c77957575-k7wx6\" (UID: \"0b83dded-aa86-4220-939a-e62e9b913f1e\") " pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.560210 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.560171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-k7wx6"
Jul 06 20:59:34.693424 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:34.693287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-k7wx6"]
Jul 06 20:59:34.695959 ip-10-0-140-38 kubenswrapper[2578]: W0706 20:59:34.695927    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b83dded_aa86_4220_939a_e62e9b913f1e.slice/crio-e0939b70ec18a5ef7d93336e086bb6e9283144013af6f8dad42986947defa15b WatchSource:0}: Error finding container e0939b70ec18a5ef7d93336e086bb6e9283144013af6f8dad42986947defa15b: Status 404 returned error can't find the container with id e0939b70ec18a5ef7d93336e086bb6e9283144013af6f8dad42986947defa15b
Jul 06 20:59:35.446791 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:35.446671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-k7wx6" event={"ID":"0b83dded-aa86-4220-939a-e62e9b913f1e","Type":"ContainerStarted","Data":"ad02030881b415032030888bad326c088c714441ff6c5ca379d3a4516cdf464a"}
Jul 06 20:59:35.446791 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:35.446744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-k7wx6" event={"ID":"0b83dded-aa86-4220-939a-e62e9b913f1e","Type":"ContainerStarted","Data":"e0939b70ec18a5ef7d93336e086bb6e9283144013af6f8dad42986947defa15b"}
Jul 06 20:59:35.466298 ip-10-0-140-38 kubenswrapper[2578]: I0706 20:59:35.466239    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-k7wx6" podStartSLOduration=0.985640706 podStartE2EDuration="1.466222139s" totalImagesPullingTime="480.581433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:35.464789883 +0000 UTC m=+617.924551760" watchObservedRunningTime="2026-07-06 20:59:35.466222139 +0000 UTC m=+617.925984027"
Jul 06 21:00:14.261170 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:00:14.261132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:15.577935 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:00:15.577898    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:19.434191 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:01:19.434111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:19.555003 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:01:19.554972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:35.536329 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:01:35.536290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:50.164511 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:01:50.164480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:50.245405 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:01:50.245370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:23.985844 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:23.985796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:24.416938 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:24.416890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:30.530669 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.530620    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"]
Jul 06 21:02:30.557177 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.557130    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"]
Jul 06 21:02:30.557336 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.557247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.560343 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.560314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:02:30.560627 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.560607    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:02:30.560751 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.560631    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 06 21:02:30.560751 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.560705    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:02:30.560876 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.560792    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:02:30.566274 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.566249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:30.742659 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.742608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-home\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.742886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.742690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-model-cache\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.742886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.742750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-dshm\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.742886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.742786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-kserve-provision-location\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.742886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.742816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmvb\" (UniqueName: \"kubernetes.io/projected/0f29c373-b490-44b0-a17f-9a26f73d2826-kube-api-access-scmvb\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.743111 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.742893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-tmp-dir\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.743111 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.742917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f29c373-b490-44b0-a17f-9a26f73d2826-tls-certs\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.844132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-tmp-dir\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.844191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f29c373-b490-44b0-a17f-9a26f73d2826-tls-certs\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.844236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-home\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.844280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-model-cache\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.844315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-dshm\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.844347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-kserve-provision-location\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.844379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmvb\" (UniqueName: \"kubernetes.io/projected/0f29c373-b490-44b0-a17f-9a26f73d2826-kube-api-access-scmvb\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.845163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-tmp-dir\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.845885    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-model-cache\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.846080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-home\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.846793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.846468    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-kserve-provision-location\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.848886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.848860    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f29c373-b490-44b0-a17f-9a26f73d2826-tls-certs\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.849749 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.849698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:30.849865 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.849799    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-dshm\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.861308 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.861287    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmvb\" (UniqueName: \"kubernetes.io/projected/0f29c373-b490-44b0-a17f-9a26f73d2826-kube-api-access-scmvb\") pod \"auth-enabled-test-kserve-7bfb886698-zqrv5\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:30.868130 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:30.868110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:02:31.050053 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:31.049988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"]
Jul 06 21:02:31.053824 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:02:31.053799    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f29c373_b490_44b0_a17f_9a26f73d2826.slice/crio-599aca9ee9971111f0a282fe32e8db89a471199b51b8ecdaf8f78f526edc7a0f WatchSource:0}: Error finding container 599aca9ee9971111f0a282fe32e8db89a471199b51b8ecdaf8f78f526edc7a0f: Status 404 returned error can't find the container with id 599aca9ee9971111f0a282fe32e8db89a471199b51b8ecdaf8f78f526edc7a0f
Jul 06 21:02:31.055694 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:31.055679    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:02:31.174423 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:31.174335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" event={"ID":"0f29c373-b490-44b0-a17f-9a26f73d2826","Type":"ContainerStarted","Data":"599aca9ee9971111f0a282fe32e8db89a471199b51b8ecdaf8f78f526edc7a0f"}
Jul 06 21:02:35.195659 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:35.195626    2578 generic.go:356] "Generic (PLEG): container finished" podID="07976c37-1aab-44af-ab95-9824a1a523b4" containerID="2bf489327374fe1de0487cf78a460fb52a00218459251dcb4eb06de8b3cb2a47" exitCode=0
Jul 06 21:02:35.196176 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:35.195698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" event={"ID":"07976c37-1aab-44af-ab95-9824a1a523b4","Type":"ContainerDied","Data":"2bf489327374fe1de0487cf78a460fb52a00218459251dcb4eb06de8b3cb2a47"}
Jul 06 21:02:35.196176 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:35.196126    2578 scope.go:117] "RemoveContainer" containerID="2bf489327374fe1de0487cf78a460fb52a00218459251dcb4eb06de8b3cb2a47"
Jul 06 21:02:35.197088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:35.197050    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:02:35.197209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:35.197132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" event={"ID":"0f29c373-b490-44b0-a17f-9a26f73d2826","Type":"ContainerStarted","Data":"3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128"}
Jul 06 21:02:35.197209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:35.197144    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:02:35.197209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:35.197183    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:02:36.204310 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:36.204269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" event={"ID":"07976c37-1aab-44af-ab95-9824a1a523b4","Type":"ContainerStarted","Data":"cbd766ab3d46fd63111c2ee42bf3e4ce2d280ca0040c98fa742145b5054f96d1"}
Jul 06 21:02:37.025530 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:37.025439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:02:37.188250 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:37.188210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:02:37.189439 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:37.189408    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:02:37.189578 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:37.189478    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:02:38.187812 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:38.187780    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:02:38.188187 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:38.187833    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:02:38.237411 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:38.237374    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerID="3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128" exitCode=0
Jul 06 21:02:38.237711 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:38.237446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" event={"ID":"0f29c373-b490-44b0-a17f-9a26f73d2826","Type":"ContainerDied","Data":"3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128"}
Jul 06 21:02:39.188768 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:39.188512    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:02:39.188768 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:39.188577    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:02:39.516391 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:39.516105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:39.572000 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:39.571966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:40.189182 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:40.188785    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:02:40.189182 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:40.188886    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:02:40.278089 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:40.277966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:41.188657 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:41.188620    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:02:41.188885 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:41.188671    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:02:42.547260 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:42.547011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:02:42.550982 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:42.550928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:02:46.318559 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:02:46.318518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:03:07.411480 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:07.411436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" event={"ID":"0f29c373-b490-44b0-a17f-9a26f73d2826","Type":"ContainerStarted","Data":"61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86"}
Jul 06 21:03:07.445311 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:07.445249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podStartSLOduration=2.381085602 podStartE2EDuration="37.445228025s" totalImagesPullingTime="31.503958376s" totalInitContainerRuntime="3.560184047s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:02:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:03:07.444395383 +0000 UTC m=+829.904157258" watchObservedRunningTime="2026-07-06 21:03:07.445228025 +0000 UTC m=+829.904989901"
Jul 06 21:03:07.868301 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:07.868258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:03:10.869077 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:10.869038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:03:10.870542 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:10.870502    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:03:20.868887 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:20.868844    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:03:30.869298 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:30.869256    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:03:36.646678 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:36.646637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:03:36.755252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:36.755217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:03:40.868807 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:40.868764    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:03:50.868794 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:03:50.868745    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:04:00.869189 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:00.869133    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:04:07.911054 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:07.911014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:09.436998 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:09.436917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:09.758177 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:09.758086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:10.869533 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:10.869469    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:04:19.438644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:19.438604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:20.869360 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:20.869310    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:04:30.868617 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:30.868547    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:04:40.868814 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:40.868759    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:04:43.315826 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:43.315793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:43.411597 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:43.411563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:50.868651 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:04:50.868602    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": dial tcp 10.134.0.25:8000: connect: connection refused"
Jul 06 21:05:00.878378 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:00.878347    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:05:00.886246 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:00.886223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:05:17.381061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:17.381028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:18.910647 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:18.910604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:20.164756 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:20.164703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:20.190963 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:20.190931    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"]
Jul 06 21:05:20.267046 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:20.267010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:30.279906 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:30.279869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:35.214934 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:35.214866    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" containerID="cri-o://61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86" gracePeriod=45
Jul 06 21:05:35.966761 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:35.966738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:05:36.034599 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f29c373-b490-44b0-a17f-9a26f73d2826-tls-certs\") pod \"0f29c373-b490-44b0-a17f-9a26f73d2826\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") "
Jul 06 21:05:36.034807 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-kserve-provision-location\") pod \"0f29c373-b490-44b0-a17f-9a26f73d2826\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") "
Jul 06 21:05:36.034807 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-home\") pod \"0f29c373-b490-44b0-a17f-9a26f73d2826\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") "
Jul 06 21:05:36.034807 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-model-cache\") pod \"0f29c373-b490-44b0-a17f-9a26f73d2826\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") "
Jul 06 21:05:36.034807 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-tmp-dir\") pod \"0f29c373-b490-44b0-a17f-9a26f73d2826\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") "
Jul 06 21:05:36.034807 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034804    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scmvb\" (UniqueName: \"kubernetes.io/projected/0f29c373-b490-44b0-a17f-9a26f73d2826-kube-api-access-scmvb\") pod \"0f29c373-b490-44b0-a17f-9a26f73d2826\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") "
Jul 06 21:05:36.035067 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-dshm\") pod \"0f29c373-b490-44b0-a17f-9a26f73d2826\" (UID: \"0f29c373-b490-44b0-a17f-9a26f73d2826\") "
Jul 06 21:05:36.035067 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.034969    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f29c373-b490-44b0-a17f-9a26f73d2826" (UID: "0f29c373-b490-44b0-a17f-9a26f73d2826"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:36.035165 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.035123    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:05:36.035526 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.035484    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-home" (OuterVolumeSpecName: "home") pod "0f29c373-b490-44b0-a17f-9a26f73d2826" (UID: "0f29c373-b490-44b0-a17f-9a26f73d2826"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:36.038060 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.038028    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-dshm" (OuterVolumeSpecName: "dshm") pod "0f29c373-b490-44b0-a17f-9a26f73d2826" (UID: "0f29c373-b490-44b0-a17f-9a26f73d2826"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:36.038060 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.038039    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f29c373-b490-44b0-a17f-9a26f73d2826-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f29c373-b490-44b0-a17f-9a26f73d2826" (UID: "0f29c373-b490-44b0-a17f-9a26f73d2826"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:05:36.038268 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.038233    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f29c373-b490-44b0-a17f-9a26f73d2826-kube-api-access-scmvb" (OuterVolumeSpecName: "kube-api-access-scmvb") pod "0f29c373-b490-44b0-a17f-9a26f73d2826" (UID: "0f29c373-b490-44b0-a17f-9a26f73d2826"). InnerVolumeSpecName "kube-api-access-scmvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:05:36.054597 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.054550    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f29c373-b490-44b0-a17f-9a26f73d2826" (UID: "0f29c373-b490-44b0-a17f-9a26f73d2826"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:36.081426 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.081392    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerID="61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86" exitCode=0
Jul 06 21:05:36.081569 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.081446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" event={"ID":"0f29c373-b490-44b0-a17f-9a26f73d2826","Type":"ContainerDied","Data":"61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86"}
Jul 06 21:05:36.081569 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.081485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" event={"ID":"0f29c373-b490-44b0-a17f-9a26f73d2826","Type":"ContainerDied","Data":"599aca9ee9971111f0a282fe32e8db89a471199b51b8ecdaf8f78f526edc7a0f"}
Jul 06 21:05:36.081569 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.081512    2578 scope.go:117] "RemoveContainer" containerID="61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86"
Jul 06 21:05:36.081569 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.081450    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"
Jul 06 21:05:36.091370 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.091345    2578 scope.go:117] "RemoveContainer" containerID="3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128"
Jul 06 21:05:36.094529 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.094501    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f29c373-b490-44b0-a17f-9a26f73d2826" (UID: "0f29c373-b490-44b0-a17f-9a26f73d2826"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:36.099043 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.099024    2578 scope.go:117] "RemoveContainer" containerID="61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86"
Jul 06 21:05:36.099301 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:05:36.099281    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86\": container with ID starting with 61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86 not found: ID does not exist" containerID="61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86"
Jul 06 21:05:36.099359 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.099311    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86"} err="failed to get container status \"61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86\": rpc error: code = NotFound desc = could not find container \"61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86\": container with ID starting with 61def427dc50ef2340c38b6292f977078adfbd418ffa18480000ee921dd25b86 not found: ID does not exist"
Jul 06 21:05:36.099359 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.099334    2578 scope.go:117] "RemoveContainer" containerID="3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128"
Jul 06 21:05:36.099594 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:05:36.099576    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128\": container with ID starting with 3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128 not found: ID does not exist" containerID="3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128"
Jul 06 21:05:36.099644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.099599    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128"} err="failed to get container status \"3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128\": rpc error: code = NotFound desc = could not find container \"3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128\": container with ID starting with 3178856f4f79b379a37170ebd8e4c6f7368ebe3adaee99013861ad4e573cb128 not found: ID does not exist"
Jul 06 21:05:36.136116 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.136086    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:05:36.136116 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.136113    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:05:36.136276 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.136123    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmvb\" (UniqueName: \"kubernetes.io/projected/0f29c373-b490-44b0-a17f-9a26f73d2826-kube-api-access-scmvb\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:05:36.136276 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.136133    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:05:36.136276 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.136142    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f29c373-b490-44b0-a17f-9a26f73d2826-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:05:36.136276 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.136150    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f29c373-b490-44b0-a17f-9a26f73d2826-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:05:36.416557 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.416472    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"]
Jul 06 21:05:36.420241 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.420215    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5"]
Jul 06 21:05:36.879316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:36.879270    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:05:38.159412 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:38.159377    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" path="/var/lib/kubelet/pods/0f29c373-b490-44b0-a17f-9a26f73d2826/volumes"
Jul 06 21:05:41.932091 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.932004    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"]
Jul 06 21:05:41.932545 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.932472    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="storage-initializer"
Jul 06 21:05:41.932545 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.932493    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="storage-initializer"
Jul 06 21:05:41.932545 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.932511    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main"
Jul 06 21:05:41.932545 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.932519    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main"
Jul 06 21:05:41.932775 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.932598    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f29c373-b490-44b0-a17f-9a26f73d2826" containerName="main"
Jul 06 21:05:41.934657 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.934635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:41.937897 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.937866    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:05:41.938623 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.938601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 06 21:05:41.948650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.948628    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:05:41.948835 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.948817    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:05:41.949413 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.949398    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:05:41.950376 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.950357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:41.965681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.965653    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"]
Jul 06 21:05:41.986219 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.986191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4ml\" (UniqueName: \"kubernetes.io/projected/ed10f754-0e69-4dbd-85b9-2f541703d53b-kube-api-access-8b4ml\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:41.986376 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.986235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-tmp-dir\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:41.986376 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.986254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-dshm\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:41.986376 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.986270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10f754-0e69-4dbd-85b9-2f541703d53b-tls-certs\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:41.986516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.986373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-home\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:41.986516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.986429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-model-cache\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:41.986516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:41.986452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.087934 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.087892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-model-cache\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.087934 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.087935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088196 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.087995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4ml\" (UniqueName: \"kubernetes.io/projected/ed10f754-0e69-4dbd-85b9-2f541703d53b-kube-api-access-8b4ml\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088196 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-tmp-dir\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088196 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-dshm\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088196 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10f754-0e69-4dbd-85b9-2f541703d53b-tls-certs\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088196 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-home\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088450 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088409    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-model-cache\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088505 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088410    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088505 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-tmp-dir\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.088684 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.088665    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-home\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.090536 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.090509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-dshm\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.090646 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.090534    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10f754-0e69-4dbd-85b9-2f541703d53b-tls-certs\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.101007 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.100979    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4ml\" (UniqueName: \"kubernetes.io/projected/ed10f754-0e69-4dbd-85b9-2f541703d53b-kube-api-access-8b4ml\") pod \"auth-invalid-token-test-kserve-75cc88cd9d-vn9hw\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.245049 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.244952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:42.313904 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.313833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:42.399065 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:42.399040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"]
Jul 06 21:05:42.400268 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:05:42.400240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded10f754_0e69_4dbd_85b9_2f541703d53b.slice/crio-fd5e862b142f7829dde439b73214298be6fd73622858d86f686963a8e0525ad2 WatchSource:0}: Error finding container fd5e862b142f7829dde439b73214298be6fd73622858d86f686963a8e0525ad2: Status 404 returned error can't find the container with id fd5e862b142f7829dde439b73214298be6fd73622858d86f686963a8e0525ad2
Jul 06 21:05:43.115166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:43.115121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" event={"ID":"ed10f754-0e69-4dbd-85b9-2f541703d53b","Type":"ContainerStarted","Data":"a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8"}
Jul 06 21:05:43.115657 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:43.115196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" event={"ID":"ed10f754-0e69-4dbd-85b9-2f541703d53b","Type":"ContainerStarted","Data":"fd5e862b142f7829dde439b73214298be6fd73622858d86f686963a8e0525ad2"}
Jul 06 21:05:46.130668 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:46.130638    2578 generic.go:356] "Generic (PLEG): container finished" podID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerID="a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8" exitCode=0
Jul 06 21:05:46.131144 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:46.130736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" event={"ID":"ed10f754-0e69-4dbd-85b9-2f541703d53b","Type":"ContainerDied","Data":"a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8"}
Jul 06 21:05:47.138376 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:47.138337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" event={"ID":"ed10f754-0e69-4dbd-85b9-2f541703d53b","Type":"ContainerStarted","Data":"94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0"}
Jul 06 21:05:47.245828 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:47.245783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:49.143879 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:49.143832    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:52.245561 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:52.245529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:05:52.247466 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:05:52.247437    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:06:02.245945 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:02.245903    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:06:10.614638 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:10.614603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:11.038122 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:11.038085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:12.245544 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:12.245480    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:06:13.460481 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:13.460441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:21.026729 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:21.026680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:22.245986 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:22.245938    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:06:32.245601 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:32.245560    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:06:42.245423 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:42.245379    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:06:44.850523 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:44.850485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:44.948486 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:44.948445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:48.906232 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:48.906195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:52.246125 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:06:52.246083    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:07:02.246126 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:02.246074    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:07:04.595126 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:04.595087    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:07:04.684199 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:04.684157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:07:12.246380 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:12.246270    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:07:22.245776 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:22.245706    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:07:28.623321 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:28.623280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:07:28.994333 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:28.994235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:07:32.245563 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:32.245519    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:07:34.090025 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:34.089985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:07:42.255409 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:42.255377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:07:42.263336 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:42.263292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:07:42.296528 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:42.296470    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podStartSLOduration=117.84724234 podStartE2EDuration="2m1.296455979s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.449213639s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:05:47.173286362 +0000 UTC m=+989.633048239" watchObservedRunningTime="2026-07-06 21:07:42.296455979 +0000 UTC m=+1104.756217855"
Jul 06 21:07:43.796554 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:43.796506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:07:43.821048 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:43.821014    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"]
Jul 06 21:07:43.861133 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:43.861098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:07:58.850958 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:58.850914    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" containerID="cri-o://94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0" gracePeriod=45
Jul 06 21:07:59.256610 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.256515    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": dial tcp 10.134.0.26:8000: connect: connection refused"
Jul 06 21:07:59.620094 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.620064    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:07:59.701168 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10f754-0e69-4dbd-85b9-2f541703d53b-tls-certs\") pod \"ed10f754-0e69-4dbd-85b9-2f541703d53b\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") "
Jul 06 21:07:59.701328 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-kserve-provision-location\") pod \"ed10f754-0e69-4dbd-85b9-2f541703d53b\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") "
Jul 06 21:07:59.701328 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701244    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-dshm\") pod \"ed10f754-0e69-4dbd-85b9-2f541703d53b\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") "
Jul 06 21:07:59.701328 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-tmp-dir\") pod \"ed10f754-0e69-4dbd-85b9-2f541703d53b\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") "
Jul 06 21:07:59.701328 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b4ml\" (UniqueName: \"kubernetes.io/projected/ed10f754-0e69-4dbd-85b9-2f541703d53b-kube-api-access-8b4ml\") pod \"ed10f754-0e69-4dbd-85b9-2f541703d53b\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") "
Jul 06 21:07:59.701546 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-model-cache\") pod \"ed10f754-0e69-4dbd-85b9-2f541703d53b\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") "
Jul 06 21:07:59.701546 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-home\") pod \"ed10f754-0e69-4dbd-85b9-2f541703d53b\" (UID: \"ed10f754-0e69-4dbd-85b9-2f541703d53b\") "
Jul 06 21:07:59.701672 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.701637    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed10f754-0e69-4dbd-85b9-2f541703d53b" (UID: "ed10f754-0e69-4dbd-85b9-2f541703d53b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:59.702170 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.702145    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-home" (OuterVolumeSpecName: "home") pod "ed10f754-0e69-4dbd-85b9-2f541703d53b" (UID: "ed10f754-0e69-4dbd-85b9-2f541703d53b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:59.703568 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.703530    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed10f754-0e69-4dbd-85b9-2f541703d53b-kube-api-access-8b4ml" (OuterVolumeSpecName: "kube-api-access-8b4ml") pod "ed10f754-0e69-4dbd-85b9-2f541703d53b" (UID: "ed10f754-0e69-4dbd-85b9-2f541703d53b"). InnerVolumeSpecName "kube-api-access-8b4ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:07:59.703676 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.703640    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-dshm" (OuterVolumeSpecName: "dshm") pod "ed10f754-0e69-4dbd-85b9-2f541703d53b" (UID: "ed10f754-0e69-4dbd-85b9-2f541703d53b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:59.703752 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.703741    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed10f754-0e69-4dbd-85b9-2f541703d53b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed10f754-0e69-4dbd-85b9-2f541703d53b" (UID: "ed10f754-0e69-4dbd-85b9-2f541703d53b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:07:59.713472 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.713437    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed10f754-0e69-4dbd-85b9-2f541703d53b" (UID: "ed10f754-0e69-4dbd-85b9-2f541703d53b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:59.722449 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.722420    2578 generic.go:356] "Generic (PLEG): container finished" podID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerID="94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0" exitCode=0
Jul 06 21:07:59.722574 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.722470    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"
Jul 06 21:07:59.722574 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.722508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" event={"ID":"ed10f754-0e69-4dbd-85b9-2f541703d53b","Type":"ContainerDied","Data":"94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0"}
Jul 06 21:07:59.722644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.722598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw" event={"ID":"ed10f754-0e69-4dbd-85b9-2f541703d53b","Type":"ContainerDied","Data":"fd5e862b142f7829dde439b73214298be6fd73622858d86f686963a8e0525ad2"}
Jul 06 21:07:59.722644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.722622    2578 scope.go:117] "RemoveContainer" containerID="94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0"
Jul 06 21:07:59.731031 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.731009    2578 scope.go:117] "RemoveContainer" containerID="a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8"
Jul 06 21:07:59.738901 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.738882    2578 scope.go:117] "RemoveContainer" containerID="94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0"
Jul 06 21:07:59.739190 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:07:59.739167    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0\": container with ID starting with 94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0 not found: ID does not exist" containerID="94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0"
Jul 06 21:07:59.739263 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.739200    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0"} err="failed to get container status \"94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0\": rpc error: code = NotFound desc = could not find container \"94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0\": container with ID starting with 94a106dea2da677adc4eb98298b930c18ada202a3f8a0dfea5749f05fbe895a0 not found: ID does not exist"
Jul 06 21:07:59.739263 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.739220    2578 scope.go:117] "RemoveContainer" containerID="a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8"
Jul 06 21:07:59.739437 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:07:59.739422    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8\": container with ID starting with a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8 not found: ID does not exist" containerID="a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8"
Jul 06 21:07:59.739483 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.739441    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8"} err="failed to get container status \"a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8\": rpc error: code = NotFound desc = could not find container \"a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8\": container with ID starting with a13dd97779bec40eacb57882bffe14c7d995c80c8b7ef88b0a2698935ef183b8 not found: ID does not exist"
Jul 06 21:07:59.765125 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.765074    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed10f754-0e69-4dbd-85b9-2f541703d53b" (UID: "ed10f754-0e69-4dbd-85b9-2f541703d53b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:59.803062 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.803024    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:07:59.803062 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.803055    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:07:59.803062 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.803063    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10f754-0e69-4dbd-85b9-2f541703d53b-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:07:59.803285 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.803072    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:07:59.803285 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.803081    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:07:59.803285 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.803090    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed10f754-0e69-4dbd-85b9-2f541703d53b-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:07:59.803285 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:07:59.803097    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4ml\" (UniqueName: \"kubernetes.io/projected/ed10f754-0e69-4dbd-85b9-2f541703d53b-kube-api-access-8b4ml\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:08:00.050852 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:00.050806    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"]
Jul 06 21:08:00.057879 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:00.057812    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw"]
Jul 06 21:08:00.159436 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:00.159400    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" path="/var/lib/kubelet/pods/ed10f754-0e69-4dbd-85b9-2f541703d53b/volumes"
Jul 06 21:08:12.675893 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.675856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:12.885243 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.885205    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"]
Jul 06 21:08:12.885490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.885479    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main"
Jul 06 21:08:12.885537 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.885492    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main"
Jul 06 21:08:12.885537 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.885510    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="storage-initializer"
Jul 06 21:08:12.885537 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.885516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="storage-initializer"
Jul 06 21:08:12.885625 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.885566    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed10f754-0e69-4dbd-85b9-2f541703d53b" containerName="main"
Jul 06 21:08:12.889138 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.889121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:12.894616 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.894488    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:08:12.894616 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.894498    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:08:12.894616 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.894551    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:08:12.895001 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.894554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-5mxzc\""
Jul 06 21:08:12.895001 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.894682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 06 21:08:12.906074 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.906047    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"]
Jul 06 21:08:12.917972 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:12.917931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:13.012739 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.012627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqzt\" (UniqueName: \"kubernetes.io/projected/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-kube-api-access-nzqzt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.012739 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.012690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.013001 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.012763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.113918 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.113884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.114096 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.113940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqzt\" (UniqueName: \"kubernetes.io/projected/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-kube-api-access-nzqzt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.114096 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.113990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.114362 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.114344    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.116411 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.116390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.123989 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.123963    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqzt\" (UniqueName: \"kubernetes.io/projected/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-kube-api-access-nzqzt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.198276 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.198234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:13.329340 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.329314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"]
Jul 06 21:08:13.331657 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:08:13.331626    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b21d5c7_6318_4a22_bbdb_c8c5892dc6f4.slice/crio-d4499862fb9ef6912ec55a99a5343aeece2545a6a25417acb52ae220dae6b2e1 WatchSource:0}: Error finding container d4499862fb9ef6912ec55a99a5343aeece2545a6a25417acb52ae220dae6b2e1: Status 404 returned error can't find the container with id d4499862fb9ef6912ec55a99a5343aeece2545a6a25417acb52ae220dae6b2e1
Jul 06 21:08:13.333637 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.333620    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:08:13.785640 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:13.785600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" event={"ID":"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4","Type":"ContainerStarted","Data":"d4499862fb9ef6912ec55a99a5343aeece2545a6a25417acb52ae220dae6b2e1"}
Jul 06 21:08:15.796614 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:15.796580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" event={"ID":"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4","Type":"ContainerStarted","Data":"188f03458e412f16374074cd79b9719f321b5823516a57401331b35c9e1d1cfb"}
Jul 06 21:08:15.797029 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:15.796648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:15.822599 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:15.819404    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" podStartSLOduration=2.260726648 podStartE2EDuration="3.819382883s" totalImagesPullingTime="1.558656235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:15.816277531 +0000 UTC m=+1138.276039437" watchObservedRunningTime="2026-07-06 21:08:15.819382883 +0000 UTC m=+1138.279144760"
Jul 06 21:08:44.710892 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.710812    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"]
Jul 06 21:08:44.718228 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.718198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.721107 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.721086    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 06 21:08:44.721246 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.721092    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:08:44.726199 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.726170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"]
Jul 06 21:08:44.738261 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.738234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:44.786443 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.786409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tmp-dir\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.786650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.786450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzg5x\" (UniqueName: \"kubernetes.io/projected/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kube-api-access-dzg5x\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.786650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.786492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tls-certs\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.786650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.786565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-home\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.786650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.786601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-model-cache\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.786650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.786647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-dshm\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.786889 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.786667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kserve-provision-location\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.887435 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.887380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tls-certs\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.887435 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.887437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-home\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.887435 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.887458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-model-cache\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.887782 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.887496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-dshm\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.887782 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.887515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kserve-provision-location\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.887782 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.887576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tmp-dir\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.887782 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.887593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzg5x\" (UniqueName: \"kubernetes.io/projected/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kube-api-access-dzg5x\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.888043 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.888019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-home\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.888104 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.888058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kserve-provision-location\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.888140 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.888097    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tmp-dir\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.888173 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.888153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-model-cache\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.889827 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.889801    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-dshm\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.890113 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.890093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tls-certs\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:44.897344 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:44.897319    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzg5x\" (UniqueName: \"kubernetes.io/projected/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kube-api-access-dzg5x\") pod \"auth-disabled-test-kserve-7d4d7b596d-74pqp\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:45.029174 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:45.029143    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:45.174244 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:45.174212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"]
Jul 06 21:08:45.176915 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:08:45.176880    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42566cd3_9021_4aa7_8f1d_289cf7dd545f.slice/crio-66d5d6401f7fde143db06c05046783d289e11b6ce6d75a87e68b1cd2aaa5cd64 WatchSource:0}: Error finding container 66d5d6401f7fde143db06c05046783d289e11b6ce6d75a87e68b1cd2aaa5cd64: Status 404 returned error can't find the container with id 66d5d6401f7fde143db06c05046783d289e11b6ce6d75a87e68b1cd2aaa5cd64
Jul 06 21:08:45.218840 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:45.218811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:45.929758 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:45.929707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" event={"ID":"42566cd3-9021-4aa7-8f1d-289cf7dd545f","Type":"ContainerStarted","Data":"0ee38d43452f6a22a431ef109207ea1cae32119e9d9b5d2e5af1e09995c2f749"}
Jul 06 21:08:45.930201 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:45.929782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" event={"ID":"42566cd3-9021-4aa7-8f1d-289cf7dd545f","Type":"ContainerStarted","Data":"66d5d6401f7fde143db06c05046783d289e11b6ce6d75a87e68b1cd2aaa5cd64"}
Jul 06 21:08:46.803319 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:46.803283    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:08:49.950566 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:49.950463    2578 generic.go:356] "Generic (PLEG): container finished" podID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerID="0ee38d43452f6a22a431ef109207ea1cae32119e9d9b5d2e5af1e09995c2f749" exitCode=0
Jul 06 21:08:49.950566 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:49.950534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" event={"ID":"42566cd3-9021-4aa7-8f1d-289cf7dd545f","Type":"ContainerDied","Data":"0ee38d43452f6a22a431ef109207ea1cae32119e9d9b5d2e5af1e09995c2f749"}
Jul 06 21:08:50.960083 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:50.960040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" event={"ID":"42566cd3-9021-4aa7-8f1d-289cf7dd545f","Type":"ContainerStarted","Data":"0cc0aed2c89e0b403bc204646e055eec58f3b25c89d5567793928c387ecf8bb4"}
Jul 06 21:08:50.984537 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:50.984482    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podStartSLOduration=2.611202989 podStartE2EDuration="6.98446707s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.373264081s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:50.983218868 +0000 UTC m=+1173.442980768" watchObservedRunningTime="2026-07-06 21:08:50.98446707 +0000 UTC m=+1173.444228946"
Jul 06 21:08:51.029302 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:51.029255    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:55.030340 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:55.030303    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:08:55.032028 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:08:55.031998    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:09:01.752033 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:01.751995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:09:05.029823 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:05.029779    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:09:15.029930 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:15.029873    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:09:18.757193 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:18.757156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:09:20.807731 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:20.807687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:09:20.824738 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:20.824684    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"]
Jul 06 21:09:20.835692 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:09:20.835649    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:09:20.835692 ip-10-0-140-38 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:09:20.835692 ip-10-0-140-38 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 06 21:09:20.835692 ip-10-0-140-38 kubenswrapper[2578]:  > containerID="188f03458e412f16374074cd79b9719f321b5823516a57401331b35c9e1d1cfb" cmd=["/bin/sleep","15"]
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:20.835734    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" message="" err=<
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]:  >
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:09:20.835754    2578 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" podUID="1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" containerName="main" containerID="cri-o://188f03458e412f16374074cd79b9719f321b5823516a57401331b35c9e1d1cfb"
Jul 06 21:09:20.835970 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:20.835802    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" podUID="1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" containerName="main" containerID="cri-o://188f03458e412f16374074cd79b9719f321b5823516a57401331b35c9e1d1cfb" gracePeriod=60
Jul 06 21:09:20.881869 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:20.881822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:09:21.096574 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.096487    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" containerID="188f03458e412f16374074cd79b9719f321b5823516a57401331b35c9e1d1cfb" exitCode=0
Jul 06 21:09:21.096574 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.096557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" event={"ID":"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4","Type":"ContainerDied","Data":"188f03458e412f16374074cd79b9719f321b5823516a57401331b35c9e1d1cfb"}
Jul 06 21:09:21.122287 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.122259    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:09:21.214476 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.214431    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tls-certs\") pod \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") "
Jul 06 21:09:21.214662 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.214495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzqzt\" (UniqueName: \"kubernetes.io/projected/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-kube-api-access-nzqzt\") pod \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") "
Jul 06 21:09:21.214662 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.214585    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tokenizer-uds\") pod \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\" (UID: \"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4\") "
Jul 06 21:09:21.214962 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.214924    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" (UID: "1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:09:21.216659 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.216636    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" (UID: "1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:09:21.216759 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.216694    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-kube-api-access-nzqzt" (OuterVolumeSpecName: "kube-api-access-nzqzt") pod "1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" (UID: "1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4"). InnerVolumeSpecName "kube-api-access-nzqzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:09:21.315446 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.315401    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:09:21.315446 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.315447    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqzt\" (UniqueName: \"kubernetes.io/projected/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-kube-api-access-nzqzt\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:09:21.315645 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:21.315465    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4-tokenizer-uds\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:09:22.104184 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:22.104150    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"
Jul 06 21:09:22.104613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:22.104156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk" event={"ID":"1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4","Type":"ContainerDied","Data":"d4499862fb9ef6912ec55a99a5343aeece2545a6a25417acb52ae220dae6b2e1"}
Jul 06 21:09:22.104613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:22.104287    2578 scope.go:117] "RemoveContainer" containerID="188f03458e412f16374074cd79b9719f321b5823516a57401331b35c9e1d1cfb"
Jul 06 21:09:22.127983 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:22.127945    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"]
Jul 06 21:09:22.133270 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:22.133242    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk"]
Jul 06 21:09:22.159217 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:22.159176    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" path="/var/lib/kubelet/pods/1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4/volumes"
Jul 06 21:09:25.029786 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:25.029744    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:09:35.030098 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:35.030056    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:09:40.976632 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:40.976596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:09:41.042040 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:41.042011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:09:45.030391 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:45.030354    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:09:55.030182 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:09:55.030130    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:10:05.030039 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:05.029990    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:10:15.029950 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.029853    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:10:15.930390 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.930345    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"]
Jul 06 21:10:15.930651 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.930639    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" containerName="main"
Jul 06 21:10:15.930700 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.930653    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" containerName="main"
Jul 06 21:10:15.930810 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.930734    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b21d5c7-6318-4a22-bbdb-c8c5892dc6f4" containerName="main"
Jul 06 21:10:15.932828 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.932811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:15.936206 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.936184    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 06 21:10:15.936502 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.936484    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:10:15.946743 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:15.946702    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"]
Jul 06 21:10:16.001512 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.001468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:10:16.077179 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.077179 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.077746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.077746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d573c6c-3954-4147-8ed5-035a1119f854-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.077746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.077746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d573c6c-3954-4147-8ed5-035a1119f854-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.077746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.077746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.077344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5b5\" (UniqueName: \"kubernetes.io/projected/3d573c6c-3954-4147-8ed5-035a1119f854-kube-api-access-7b5b5\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178053 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178256 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178256 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178256 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d573c6c-3954-4147-8ed5-035a1119f854-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178256 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178497 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d573c6c-3954-4147-8ed5-035a1119f854-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178497 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178497 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5b5\" (UniqueName: \"kubernetes.io/projected/3d573c6c-3954-4147-8ed5-035a1119f854-kube-api-access-7b5b5\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178659 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.178754 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178695    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.179013 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.178985    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.179193 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.179167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.179316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.179297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d573c6c-3954-4147-8ed5-035a1119f854-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.181217 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.181160    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.181592 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.181558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d573c6c-3954-4147-8ed5-035a1119f854-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.196114 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.196077    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5b5\" (UniqueName: \"kubernetes.io/projected/3d573c6c-3954-4147-8ed5-035a1119f854-kube-api-access-7b5b5\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.243849 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.243804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:16.260099 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.260054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:10:16.399519 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:16.399491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"]
Jul 06 21:10:16.401906 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:10:16.401875    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d573c6c_3954_4147_8ed5_035a1119f854.slice/crio-3ed3512f2f6fe21e37425eb83385a221c9ca60a4f9f9997a00f5601565790f4c WatchSource:0}: Error finding container 3ed3512f2f6fe21e37425eb83385a221c9ca60a4f9f9997a00f5601565790f4c: Status 404 returned error can't find the container with id 3ed3512f2f6fe21e37425eb83385a221c9ca60a4f9f9997a00f5601565790f4c
Jul 06 21:10:17.354493 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:17.354403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" event={"ID":"3d573c6c-3954-4147-8ed5-035a1119f854","Type":"ContainerStarted","Data":"c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb"}
Jul 06 21:10:17.354493 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:17.354489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" event={"ID":"3d573c6c-3954-4147-8ed5-035a1119f854","Type":"ContainerStarted","Data":"3ed3512f2f6fe21e37425eb83385a221c9ca60a4f9f9997a00f5601565790f4c"}
Jul 06 21:10:21.377229 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:21.377196    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d573c6c-3954-4147-8ed5-035a1119f854" containerID="c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb" exitCode=0
Jul 06 21:10:21.377610 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:21.377255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" event={"ID":"3d573c6c-3954-4147-8ed5-035a1119f854","Type":"ContainerDied","Data":"c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb"}
Jul 06 21:10:22.385607 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:22.385563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" event={"ID":"3d573c6c-3954-4147-8ed5-035a1119f854","Type":"ContainerStarted","Data":"cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b"}
Jul 06 21:10:22.413972 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:22.413909    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podStartSLOduration=3.480675771 podStartE2EDuration="7.413886002s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.933210231s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:10:22.411172418 +0000 UTC m=+1264.870934295" watchObservedRunningTime="2026-07-06 21:10:22.413886002 +0000 UTC m=+1264.873647877"
Jul 06 21:10:23.244225 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:23.244173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:25.029817 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:25.029766    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:10:26.244860 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:26.244819    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:10:26.246654 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:26.246615    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:10:28.877050 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:28.877015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:10:35.030227 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:35.030181    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:10:36.244689 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:36.244634    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:10:45.030084 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:45.030043    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 06 21:10:46.245276 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:46.245215    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:10:55.040298 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:55.040266    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:10:55.048410 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:55.048379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:10:56.244971 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:10:56.244927    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:11:00.033196 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:00.033160    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"]
Jul 06 21:11:00.065484 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:00.065440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:00.137444 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:00.137372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:06.245201 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:06.245154    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:11:14.669182 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:14.669149    2578 generic.go:356] "Generic (PLEG): container finished" podID="07976c37-1aab-44af-ab95-9824a1a523b4" containerID="cbd766ab3d46fd63111c2ee42bf3e4ce2d280ca0040c98fa742145b5054f96d1" exitCode=137
Jul 06 21:11:14.669613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:14.669200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" event={"ID":"07976c37-1aab-44af-ab95-9824a1a523b4","Type":"ContainerDied","Data":"cbd766ab3d46fd63111c2ee42bf3e4ce2d280ca0040c98fa742145b5054f96d1"}
Jul 06 21:11:14.669613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:14.669252    2578 scope.go:117] "RemoveContainer" containerID="2bf489327374fe1de0487cf78a460fb52a00218459251dcb4eb06de8b3cb2a47"
Jul 06 21:11:14.669713 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:14.669700    2578 scope.go:117] "RemoveContainer" containerID="cbd766ab3d46fd63111c2ee42bf3e4ce2d280ca0040c98fa742145b5054f96d1"
Jul 06 21:11:14.669973 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:11:14.669939    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_openshift-ingress(07976c37-1aab-44af-ab95-9824a1a523b4)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4"
Jul 06 21:11:15.056577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.056506    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main" containerID="cri-o://0cc0aed2c89e0b403bc204646e055eec58f3b25c89d5567793928c387ecf8bb4" gracePeriod=45
Jul 06 21:11:15.677209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.677172    2578 generic.go:356] "Generic (PLEG): container finished" podID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerID="0cc0aed2c89e0b403bc204646e055eec58f3b25c89d5567793928c387ecf8bb4" exitCode=0
Jul 06 21:11:15.677546 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.677237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" event={"ID":"42566cd3-9021-4aa7-8f1d-289cf7dd545f","Type":"ContainerDied","Data":"0cc0aed2c89e0b403bc204646e055eec58f3b25c89d5567793928c387ecf8bb4"}
Jul 06 21:11:15.799193 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.799166    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:11:15.841645 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.841560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kserve-provision-location\") pod \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") "
Jul 06 21:11:15.841645 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.841620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tmp-dir\") pod \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") "
Jul 06 21:11:15.841645 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.841646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-model-cache\") pod \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") "
Jul 06 21:11:15.841937 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.841697    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzg5x\" (UniqueName: \"kubernetes.io/projected/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kube-api-access-dzg5x\") pod \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") "
Jul 06 21:11:15.841937 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.841747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-dshm\") pod \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") "
Jul 06 21:11:15.841937 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.841783    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-home\") pod \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") "
Jul 06 21:11:15.841937 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.841808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tls-certs\") pod \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\" (UID: \"42566cd3-9021-4aa7-8f1d-289cf7dd545f\") "
Jul 06 21:11:15.842834 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.842386    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-model-cache" (OuterVolumeSpecName: "model-cache") pod "42566cd3-9021-4aa7-8f1d-289cf7dd545f" (UID: "42566cd3-9021-4aa7-8f1d-289cf7dd545f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:15.842834 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.842785    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-home" (OuterVolumeSpecName: "home") pod "42566cd3-9021-4aa7-8f1d-289cf7dd545f" (UID: "42566cd3-9021-4aa7-8f1d-289cf7dd545f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:15.844569 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.844540    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-dshm" (OuterVolumeSpecName: "dshm") pod "42566cd3-9021-4aa7-8f1d-289cf7dd545f" (UID: "42566cd3-9021-4aa7-8f1d-289cf7dd545f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:15.844569 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.844555    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42566cd3-9021-4aa7-8f1d-289cf7dd545f" (UID: "42566cd3-9021-4aa7-8f1d-289cf7dd545f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:11:15.844805 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.844652    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kube-api-access-dzg5x" (OuterVolumeSpecName: "kube-api-access-dzg5x") pod "42566cd3-9021-4aa7-8f1d-289cf7dd545f" (UID: "42566cd3-9021-4aa7-8f1d-289cf7dd545f"). InnerVolumeSpecName "kube-api-access-dzg5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:11:15.854816 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.854785    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42566cd3-9021-4aa7-8f1d-289cf7dd545f" (UID: "42566cd3-9021-4aa7-8f1d-289cf7dd545f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:15.915554 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.915489    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42566cd3-9021-4aa7-8f1d-289cf7dd545f" (UID: "42566cd3-9021-4aa7-8f1d-289cf7dd545f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:15.943506 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.943465    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:11:15.943506 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.943493    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:11:15.943506 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.943503    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:11:15.943506 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.943514    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:11:15.943890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.943524    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:11:15.943890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.943534    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzg5x\" (UniqueName: \"kubernetes.io/projected/42566cd3-9021-4aa7-8f1d-289cf7dd545f-kube-api-access-dzg5x\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:11:15.943890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:15.943541    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42566cd3-9021-4aa7-8f1d-289cf7dd545f-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:11:16.244485 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:16.244436    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:11:16.683695 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:16.683662    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"
Jul 06 21:11:16.684204 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:16.683662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp" event={"ID":"42566cd3-9021-4aa7-8f1d-289cf7dd545f","Type":"ContainerDied","Data":"66d5d6401f7fde143db06c05046783d289e11b6ce6d75a87e68b1cd2aaa5cd64"}
Jul 06 21:11:16.684204 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:16.683782    2578 scope.go:117] "RemoveContainer" containerID="0cc0aed2c89e0b403bc204646e055eec58f3b25c89d5567793928c387ecf8bb4"
Jul 06 21:11:16.691994 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:16.691975    2578 scope.go:117] "RemoveContainer" containerID="0ee38d43452f6a22a431ef109207ea1cae32119e9d9b5d2e5af1e09995c2f749"
Jul 06 21:11:16.717349 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:16.717319    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"]
Jul 06 21:11:16.723206 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:16.723178    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp"]
Jul 06 21:11:18.161444 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:18.161400    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" path="/var/lib/kubelet/pods/42566cd3-9021-4aa7-8f1d-289cf7dd545f/volumes"
Jul 06 21:11:20.756217 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:20.756179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:20.828841 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:20.828807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:21.320834 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.320797    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"]
Jul 06 21:11:21.321112 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.321100    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main"
Jul 06 21:11:21.321153 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.321114    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main"
Jul 06 21:11:21.321153 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.321125    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="storage-initializer"
Jul 06 21:11:21.321153 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.321132    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="storage-initializer"
Jul 06 21:11:21.321248 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.321191    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="42566cd3-9021-4aa7-8f1d-289cf7dd545f" containerName="main"
Jul 06 21:11:21.323920 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.323903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.326786 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.326762    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-vdr6g\""
Jul 06 21:11:21.327155 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.327139    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 06 21:11:21.342009 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.341979    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"]
Jul 06 21:11:21.391177 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391373 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-kube-api-access-75m6n\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391373 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391373 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391373 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391373 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391592 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391592 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.391592 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.391453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.491882 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.491848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-kube-api-access-75m6n\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.491917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.491939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.491997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492412 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492412 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492412 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492815 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492789    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.492913 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.492860    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.493154 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.493127    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.493329 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.493305    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.494635 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.494614    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.494785 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.494766    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.501319 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.501293    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-kube-api-access-75m6n\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.502708 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.502688    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-4sr52\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.634141 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.634031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:21.772499 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.772467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"]
Jul 06 21:11:21.773990 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:11:21.773958    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b9df5e_4d08_4f3e_996e_0282d903ebeb.slice/crio-1488f4766b6b574ac3b7c7a2e013ecbd29ea52463aa31f0c8e51264449404835 WatchSource:0}: Error finding container 1488f4766b6b574ac3b7c7a2e013ecbd29ea52463aa31f0c8e51264449404835: Status 404 returned error can't find the container with id 1488f4766b6b574ac3b7c7a2e013ecbd29ea52463aa31f0c8e51264449404835
Jul 06 21:11:21.775881 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.775831    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:21.775956 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.775934    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:21.775996 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:21.775978    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:22.717767 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:22.717735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" event={"ID":"25b9df5e-4d08-4f3e-996e-0282d903ebeb","Type":"ContainerStarted","Data":"a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3"}
Jul 06 21:11:22.717943 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:22.717793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" event={"ID":"25b9df5e-4d08-4f3e-996e-0282d903ebeb","Type":"ContainerStarted","Data":"1488f4766b6b574ac3b7c7a2e013ecbd29ea52463aa31f0c8e51264449404835"}
Jul 06 21:11:22.743205 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:22.743150    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" podStartSLOduration=1.743134314 podStartE2EDuration="1.743134314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:11:22.74213845 +0000 UTC m=+1325.201900327" watchObservedRunningTime="2026-07-06 21:11:22.743134314 +0000 UTC m=+1325.202896189"
Jul 06 21:11:23.634543 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:23.634498    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:23.639449 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:23.639419    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:23.722984 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:23.722942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:23.724027 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:23.724002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:24.574788 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:24.574750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:25.154667 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:25.154639    2578 scope.go:117] "RemoveContainer" containerID="cbd766ab3d46fd63111c2ee42bf3e4ce2d280ca0040c98fa742145b5054f96d1"
Jul 06 21:11:25.155923 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:25.155890    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:25.156032 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:25.155979    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:25.156094 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:25.156033    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:25.739860 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:25.739814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" event={"ID":"07976c37-1aab-44af-ab95-9824a1a523b4","Type":"ContainerStarted","Data":"a1efcdbe8d83cddedf58e2fb9fdbae3df6b0446a4c8cb8a6f0a1f4e68ec70be6"}
Jul 06 21:11:26.187992 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:26.187952    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:11:26.188982 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:26.188956    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:11:26.189114 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:26.189006    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:11:26.244644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:26.244594    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:11:26.898343 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:26.898295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:27.188698 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:27.188608    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:11:27.189059 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:27.188697    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:11:27.547860 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:27.547818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:11:28.188211 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:28.188169    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:11:28.188531 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:28.188496    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:11:29.188326 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:29.188284    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:11:29.188711 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:29.188360    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:11:30.188683 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:30.188642    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:11:30.189099 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:30.188712    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:11:31.188343 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:31.188308    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 06 21:11:31.188508 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:31.188373    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 06 21:11:33.189063 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:33.189020    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 06 21:11:33.189552 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:33.189115    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg" podUID="07976c37-1aab-44af-ab95-9824a1a523b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:11:33.586556 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:33.586520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:33.600848 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:33.600810    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:11:33.602295 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:33.602261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg"
Jul 06 21:11:36.245088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:36.245030    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 06 21:11:36.809999 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:36.809959    2578 generic.go:356] "Generic (PLEG): container finished" podID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerID="a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3" exitCode=0
Jul 06 21:11:36.810232 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:36.810041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" event={"ID":"25b9df5e-4d08-4f3e-996e-0282d903ebeb","Type":"ContainerDied","Data":"a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3"}
Jul 06 21:11:36.810584 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:36.810563    2578 scope.go:117] "RemoveContainer" containerID="a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3"
Jul 06 21:11:36.811832 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:36.811793    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:36.811952 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:36.811891    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:36.812033 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:36.811975    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 06 21:11:37.816555 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:37.816514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" event={"ID":"25b9df5e-4d08-4f3e-996e-0282d903ebeb","Type":"ContainerStarted","Data":"15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4"}
Jul 06 21:11:38.635306 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:38.635264    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:38.636786 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:38.636745    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 21:11:38.723937 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:38.723900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:39.634908 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:39.634802    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 21:11:40.635444 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:40.635387    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 21:11:41.635558 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:41.635505    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 21:11:42.635159 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:42.635104    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 21:11:43.639065 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:43.639029    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:43.640086 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:43.640065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:11:46.255053 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:46.255018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:11:46.262964 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:46.262935    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:11:51.205038 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:11:51.204991    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:12.272964 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:12.272928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:12.325129 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:12.325095    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"]
Jul 06 21:12:12.359817 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:12.359783    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:27.291472 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:27.291439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:27.347876 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:27.347826    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main" containerID="cri-o://cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b" gracePeriod=45
Jul 06 21:12:28.002729 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.002688    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:12:28.055133 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.055097    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d573c6c-3954-4147-8ed5-035a1119f854" containerID="cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b" exitCode=0
Jul 06 21:12:28.055294 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.055155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"
Jul 06 21:12:28.055294 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.055183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" event={"ID":"3d573c6c-3954-4147-8ed5-035a1119f854","Type":"ContainerDied","Data":"cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b"}
Jul 06 21:12:28.055294 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.055268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh" event={"ID":"3d573c6c-3954-4147-8ed5-035a1119f854","Type":"ContainerDied","Data":"3ed3512f2f6fe21e37425eb83385a221c9ca60a4f9f9997a00f5601565790f4c"}
Jul 06 21:12:28.055294 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.055294    2578 scope.go:117] "RemoveContainer" containerID="cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b"
Jul 06 21:12:28.063182 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.063165    2578 scope.go:117] "RemoveContainer" containerID="c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb"
Jul 06 21:12:28.071001 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.070982    2578 scope.go:117] "RemoveContainer" containerID="cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b"
Jul 06 21:12:28.071237 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:12:28.071216    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b\": container with ID starting with cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b not found: ID does not exist" containerID="cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b"
Jul 06 21:12:28.071292 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.071245    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b"} err="failed to get container status \"cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b\": rpc error: code = NotFound desc = could not find container \"cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b\": container with ID starting with cce3ac48b9a5a4a2504d8efc20365675522cb98a2640dab2c60f223e085e2f2b not found: ID does not exist"
Jul 06 21:12:28.071292 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.071261    2578 scope.go:117] "RemoveContainer" containerID="c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb"
Jul 06 21:12:28.071516 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:12:28.071494    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb\": container with ID starting with c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb not found: ID does not exist" containerID="c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb"
Jul 06 21:12:28.071561 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.071522    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb"} err="failed to get container status \"c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb\": rpc error: code = NotFound desc = could not find container \"c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb\": container with ID starting with c28d221e43771830cc91020899cfeb47b7cb59e1cd77f88fa415240c1bfaa2fb not found: ID does not exist"
Jul 06 21:12:28.108444 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108412    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-kserve-provision-location\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108605 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-home\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108605 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d573c6c-3954-4147-8ed5-035a1119f854-tls-certs\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108605 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108531    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-model-cache\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108605 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-dshm\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108605 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b5b5\" (UniqueName: \"kubernetes.io/projected/3d573c6c-3954-4147-8ed5-035a1119f854-kube-api-access-7b5b5\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108875 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108621    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d573c6c-3954-4147-8ed5-035a1119f854-cabundle-cert\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108875 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-tmp-dir\") pod \"3d573c6c-3954-4147-8ed5-035a1119f854\" (UID: \"3d573c6c-3954-4147-8ed5-035a1119f854\") "
Jul 06 21:12:28.108875 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108845    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:28.109018 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108971    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-home" (OuterVolumeSpecName: "home") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:28.109018 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.108955    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.109152 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.109128    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d573c6c-3954-4147-8ed5-035a1119f854-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:12:28.110696 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.110665    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-dshm" (OuterVolumeSpecName: "dshm") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:28.111016 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.110978    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d573c6c-3954-4147-8ed5-035a1119f854-kube-api-access-7b5b5" (OuterVolumeSpecName: "kube-api-access-7b5b5") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "kube-api-access-7b5b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:12:28.111286 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.111262    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d573c6c-3954-4147-8ed5-035a1119f854-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:12:28.120457 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.120423    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:28.164996 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.164964    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d573c6c-3954-4147-8ed5-035a1119f854" (UID: "3d573c6c-3954-4147-8ed5-035a1119f854"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:28.209390 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.209359    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.209390 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.209388    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.209580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.209399    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.209580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.209407    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d573c6c-3954-4147-8ed5-035a1119f854-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.209580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.209415    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d573c6c-3954-4147-8ed5-035a1119f854-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.209580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.209423    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5b5\" (UniqueName: \"kubernetes.io/projected/3d573c6c-3954-4147-8ed5-035a1119f854-kube-api-access-7b5b5\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.209580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.209431    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d573c6c-3954-4147-8ed5-035a1119f854-cabundle-cert\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:12:28.382575 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.382542    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"]
Jul 06 21:12:28.386904 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:28.386875    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh"]
Jul 06 21:12:30.160043 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:30.160012    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" path="/var/lib/kubelet/pods/3d573c6c-3954-4147-8ed5-035a1119f854/volumes"
Jul 06 21:12:35.422661 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.422620    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"]
Jul 06 21:12:35.423124 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.422939    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="storage-initializer"
Jul 06 21:12:35.423124 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.422952    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="storage-initializer"
Jul 06 21:12:35.423124 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.422976    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main"
Jul 06 21:12:35.423124 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.422981    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main"
Jul 06 21:12:35.423124 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.423032    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d573c6c-3954-4147-8ed5-035a1119f854" containerName="main"
Jul 06 21:12:35.426149 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.426132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.428795 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.428775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:12:35.428795 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.428788    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:12:35.428977 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.428791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:12:35.428977 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.428886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 06 21:12:35.436805 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.436781    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"]
Jul 06 21:12:35.454333 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.454305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:35.572304 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.572483 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f78905f-0e51-45ba-ba57-a81ff0138c97-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.572483 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.572483 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.572483 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.572622 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.572622 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7jc\" (UniqueName: \"kubernetes.io/projected/4f78905f-0e51-45ba-ba57-a81ff0138c97-kube-api-access-7h7jc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.572622 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.572555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f78905f-0e51-45ba-ba57-a81ff0138c97-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674031 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.673928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f78905f-0e51-45ba-ba57-a81ff0138c97-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674031 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.673992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f78905f-0e51-45ba-ba57-a81ff0138c97-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7jc\" (UniqueName: \"kubernetes.io/projected/4f78905f-0e51-45ba-ba57-a81ff0138c97-kube-api-access-7h7jc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674609 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674581    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674689 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674592    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674689 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674648    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.674849 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.674741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.675189 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.675167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f78905f-0e51-45ba-ba57-a81ff0138c97-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.676943 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.676913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.677076 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.677057    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f78905f-0e51-45ba-ba57-a81ff0138c97-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.681339 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.681312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:35.683561 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.683533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7jc\" (UniqueName: \"kubernetes.io/projected/4f78905f-0e51-45ba-ba57-a81ff0138c97-kube-api-access-7h7jc\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.736630 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.736593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:35.877358 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:35.877331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"]
Jul 06 21:12:35.879343 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:12:35.879318    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f78905f_0e51_45ba_ba57_a81ff0138c97.slice/crio-7afcc8435824afe7688fb16e5339f712345f9cd1f104edc748e9c024e1a8b19f WatchSource:0}: Error finding container 7afcc8435824afe7688fb16e5339f712345f9cd1f104edc748e9c024e1a8b19f: Status 404 returned error can't find the container with id 7afcc8435824afe7688fb16e5339f712345f9cd1f104edc748e9c024e1a8b19f
Jul 06 21:12:36.091473 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:36.091429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" event={"ID":"4f78905f-0e51-45ba-ba57-a81ff0138c97","Type":"ContainerStarted","Data":"90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787"}
Jul 06 21:12:36.091639 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:36.091496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" event={"ID":"4f78905f-0e51-45ba-ba57-a81ff0138c97","Type":"ContainerStarted","Data":"7afcc8435824afe7688fb16e5339f712345f9cd1f104edc748e9c024e1a8b19f"}
Jul 06 21:12:41.116123 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:41.116088    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerID="90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787" exitCode=0
Jul 06 21:12:41.116502 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:41.116161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" event={"ID":"4f78905f-0e51-45ba-ba57-a81ff0138c97","Type":"ContainerDied","Data":"90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787"}
Jul 06 21:12:42.123284 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:42.123249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" event={"ID":"4f78905f-0e51-45ba-ba57-a81ff0138c97","Type":"ContainerStarted","Data":"191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b"}
Jul 06 21:12:42.148865 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:42.148811    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podStartSLOduration=2.713581063 podStartE2EDuration="7.148794649s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.435213586s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:42.146976901 +0000 UTC m=+1404.606738788" watchObservedRunningTime="2026-07-06 21:12:42.148794649 +0000 UTC m=+1404.608556525"
Jul 06 21:12:42.737428 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:42.737392    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:45.737519 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:45.737471    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:12:45.739182 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:45.739153    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:12:52.629488 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:52.629451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:55.737077 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:12:55.737023    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:13:05.737371 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:05.737321    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:13:15.737385 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:15.737284    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:13:25.737448 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:25.737393    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:13:35.737895 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:35.737843    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:13:45.737588 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:45.737531    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:13:50.482609 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:50.482577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:13:53.006144 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:53.006105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:13:53.103025 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:53.102975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:13:55.737588 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:13:55.737544    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:14:05.748815 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:05.748782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:14:05.757176 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:05.757139    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:14:07.073244 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:07.073211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:07.088558 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:07.088521    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"]
Jul 06 21:14:07.173149 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:07.173112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:20.969091 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:20.969041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:22.565919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:22.565859    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" containerID="cri-o://191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b" gracePeriod=45
Jul 06 21:14:22.749577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:22.749529    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 06 21:14:23.126695 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.126670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:14:23.234114 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-model-cache\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234114 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h7jc\" (UniqueName: \"kubernetes.io/projected/4f78905f-0e51-45ba-ba57-a81ff0138c97-kube-api-access-7h7jc\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234114 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f78905f-0e51-45ba-ba57-a81ff0138c97-cabundle-cert\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234406 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f78905f-0e51-45ba-ba57-a81ff0138c97-tls-certs\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234406 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-kserve-provision-location\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234406 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-tmp-dir\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234406 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-dshm\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234406 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-home\") pod \"4f78905f-0e51-45ba-ba57-a81ff0138c97\" (UID: \"4f78905f-0e51-45ba-ba57-a81ff0138c97\") "
Jul 06 21:14:23.234406 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234353    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:23.234780 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234549    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f78905f-0e51-45ba-ba57-a81ff0138c97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:14:23.234780 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234652    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.234780 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234668    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f78905f-0e51-45ba-ba57-a81ff0138c97-cabundle-cert\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.235005 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234670    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:23.235005 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.234878    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-home" (OuterVolumeSpecName: "home") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:23.236393 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.236369    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f78905f-0e51-45ba-ba57-a81ff0138c97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:14:23.236393 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.236381    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-dshm" (OuterVolumeSpecName: "dshm") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:23.236599 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.236583    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f78905f-0e51-45ba-ba57-a81ff0138c97-kube-api-access-7h7jc" (OuterVolumeSpecName: "kube-api-access-7h7jc") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "kube-api-access-7h7jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:14:23.300095 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.300052    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f78905f-0e51-45ba-ba57-a81ff0138c97" (UID: "4f78905f-0e51-45ba-ba57-a81ff0138c97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:23.335597 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.335564    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7jc\" (UniqueName: \"kubernetes.io/projected/4f78905f-0e51-45ba-ba57-a81ff0138c97-kube-api-access-7h7jc\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.335597 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.335590    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f78905f-0e51-45ba-ba57-a81ff0138c97-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.335597 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.335601    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.335852 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.335629    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.335852 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.335638    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.335852 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.335646    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f78905f-0e51-45ba-ba57-a81ff0138c97-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:14:23.609666 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.609635    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerID="191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b" exitCode=0
Jul 06 21:14:23.610064 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.609692    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"
Jul 06 21:14:23.610064 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.609744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" event={"ID":"4f78905f-0e51-45ba-ba57-a81ff0138c97","Type":"ContainerDied","Data":"191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b"}
Jul 06 21:14:23.610064 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.609800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l" event={"ID":"4f78905f-0e51-45ba-ba57-a81ff0138c97","Type":"ContainerDied","Data":"7afcc8435824afe7688fb16e5339f712345f9cd1f104edc748e9c024e1a8b19f"}
Jul 06 21:14:23.610064 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.609825    2578 scope.go:117] "RemoveContainer" containerID="191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b"
Jul 06 21:14:23.618318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.618303    2578 scope.go:117] "RemoveContainer" containerID="90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787"
Jul 06 21:14:23.626494 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.626475    2578 scope.go:117] "RemoveContainer" containerID="191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b"
Jul 06 21:14:23.626781 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:14:23.626758    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b\": container with ID starting with 191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b not found: ID does not exist" containerID="191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b"
Jul 06 21:14:23.626836 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.626790    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b"} err="failed to get container status \"191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b\": rpc error: code = NotFound desc = could not find container \"191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b\": container with ID starting with 191cf2998d61b5c1dd0bd6c9363eac7f9b0e3bebcacc57ad0d9c796c99bfbd2b not found: ID does not exist"
Jul 06 21:14:23.626836 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.626809    2578 scope.go:117] "RemoveContainer" containerID="90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787"
Jul 06 21:14:23.627080 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:14:23.627060    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787\": container with ID starting with 90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787 not found: ID does not exist" containerID="90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787"
Jul 06 21:14:23.627141 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.627089    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787"} err="failed to get container status \"90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787\": rpc error: code = NotFound desc = could not find container \"90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787\": container with ID starting with 90740dede9212e8e2e477f214699667c342bb8819e335464b6168b44c7e17787 not found: ID does not exist"
Jul 06 21:14:23.633573 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.633543    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"]
Jul 06 21:14:23.640580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:23.640554    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l"]
Jul 06 21:14:24.160433 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:24.160399    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" path="/var/lib/kubelet/pods/4f78905f-0e51-45ba-ba57-a81ff0138c97/volumes"
Jul 06 21:14:25.897866 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.897833    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"]
Jul 06 21:14:25.898233 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.898185    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main"
Jul 06 21:14:25.898233 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.898200    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main"
Jul 06 21:14:25.898233 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.898216    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="storage-initializer"
Jul 06 21:14:25.898233 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.898222    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="storage-initializer"
Jul 06 21:14:25.898358 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.898274    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f78905f-0e51-45ba-ba57-a81ff0138c97" containerName="main"
Jul 06 21:14:25.901700 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.901679    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:25.904923 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.904900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:14:25.905069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.904946    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:14:25.905069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.904984    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 06 21:14:25.913290 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.913267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"]
Jul 06 21:14:25.959529 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.959484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:25.959685 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.959554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqsl\" (UniqueName: \"kubernetes.io/projected/343f8755-b17f-44d5-bba1-815638519304-kube-api-access-gbqsl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:25.959685 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.959589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:25.959685 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.959627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:25.959685 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.959652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:25.959685 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.959671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:25.959953 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:25.959756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343f8755-b17f-44d5-bba1-815638519304-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.060553 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.060514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqsl\" (UniqueName: \"kubernetes.io/projected/343f8755-b17f-44d5-bba1-815638519304-kube-api-access-gbqsl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.060787 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.060562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.060787 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.060604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.060787 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.060633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.060787 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.060659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.060787 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.060694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343f8755-b17f-44d5-bba1-815638519304-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.061075 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.060797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.061126 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.061085    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.061168 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.061116    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.061168 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.061147    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.061235 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.061196    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.063025 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.062998    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.063349 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.063327    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343f8755-b17f-44d5-bba1-815638519304-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.070294 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.070271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqsl\" (UniqueName: \"kubernetes.io/projected/343f8755-b17f-44d5-bba1-815638519304-kube-api-access-gbqsl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.212550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.212452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:26.353743 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.353694    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"]
Jul 06 21:14:26.356627 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:14:26.356594    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343f8755_b17f_44d5_bba1_815638519304.slice/crio-394db80bf7e441928af883bd7591470b6c1dd6c99f6105b51d3a07b158f03cd7 WatchSource:0}: Error finding container 394db80bf7e441928af883bd7591470b6c1dd6c99f6105b51d3a07b158f03cd7: Status 404 returned error can't find the container with id 394db80bf7e441928af883bd7591470b6c1dd6c99f6105b51d3a07b158f03cd7
Jul 06 21:14:26.625278 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.625234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" event={"ID":"343f8755-b17f-44d5-bba1-815638519304","Type":"ContainerStarted","Data":"c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129"}
Jul 06 21:14:26.625474 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.625288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" event={"ID":"343f8755-b17f-44d5-bba1-815638519304","Type":"ContainerStarted","Data":"394db80bf7e441928af883bd7591470b6c1dd6c99f6105b51d3a07b158f03cd7"}
Jul 06 21:14:26.728060 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.728023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:26.828596 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:26.828560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:27.478173 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:27.478125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:27.586351 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:27.586317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:27.696291 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:27.696255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:30.063678 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:30.063642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:30.119160 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:30.119126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:30.668415 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:30.668380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:30.732336 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:30.732296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:31.653845 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:31.653760    2578 generic.go:356] "Generic (PLEG): container finished" podID="343f8755-b17f-44d5-bba1-815638519304" containerID="c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129" exitCode=0
Jul 06 21:14:31.653845 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:31.653833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" event={"ID":"343f8755-b17f-44d5-bba1-815638519304","Type":"ContainerDied","Data":"c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129"}
Jul 06 21:14:32.659671 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:32.659638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" event={"ID":"343f8755-b17f-44d5-bba1-815638519304","Type":"ContainerStarted","Data":"10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67"}
Jul 06 21:14:32.691869 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:32.691674    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podStartSLOduration=2.80849482 podStartE2EDuration="7.691657676s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.883162856s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:32.689161051 +0000 UTC m=+1515.148922927" watchObservedRunningTime="2026-07-06 21:14:32.691657676 +0000 UTC m=+1515.151419552"
Jul 06 21:14:33.212629 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:33.212593    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:36.213266 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:36.213231    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:14:36.215173 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:36.215140    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:14:42.487063 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:42.486975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:46.213963 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:46.213912    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:14:56.213440 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.213388    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:14:56.303633 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.303596    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"]
Jul 06 21:14:56.306381 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.306363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.309043 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.309018    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 06 21:14:56.318767 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.318740    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"]
Jul 06 21:14:56.328133 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.328084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:56.435000 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.434953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56933010-65d6-45d1-8709-c2c2207b1c98-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.435204 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.435084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.435204 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.435123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.435204 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.435170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.435204 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.435198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.435385 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.435256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbmr\" (UniqueName: \"kubernetes.io/projected/56933010-65d6-45d1-8709-c2c2207b1c98-kube-api-access-djbmr\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536005 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.535970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbmr\" (UniqueName: \"kubernetes.io/projected/56933010-65d6-45d1-8709-c2c2207b1c98-kube-api-access-djbmr\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536174 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56933010-65d6-45d1-8709-c2c2207b1c98-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536174 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536174 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536174 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536174 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536495    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536591 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.536639 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.536583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.538634 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.538605    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.538969 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.538946    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56933010-65d6-45d1-8709-c2c2207b1c98-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.547344 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.547313    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbmr\" (UniqueName: \"kubernetes.io/projected/56933010-65d6-45d1-8709-c2c2207b1c98-kube-api-access-djbmr\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:56.769650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:56.769618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:59.360637 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.360594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.368581 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.368551    2578 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.461229 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.461188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.461402 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.461317    2578 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.464407 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.464387    2578 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") DevicePath \"csi-f383cc3e98112493d8b761a51a111ff1351ce11eb332292421bab9c54484e6b7\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.558872 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.558839    2578 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/695b1fedd1efd2a2620b46c540e3c4cbf5ecfc1b083102e8f2c618a1c9ec9ff5/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.576603 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.576575    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.617277 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.617188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:14:59.768001 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.767963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"]
Jul 06 21:14:59.771297 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:14:59.771260    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56933010_65d6_45d1_8709_c2c2207b1c98.slice/crio-103d1ff9a6676885b8579ed22a2709844499b8df55f459b0c1549a5baa94e26a WatchSource:0}: Error finding container 103d1ff9a6676885b8579ed22a2709844499b8df55f459b0c1549a5baa94e26a: Status 404 returned error can't find the container with id 103d1ff9a6676885b8579ed22a2709844499b8df55f459b0c1549a5baa94e26a
Jul 06 21:14:59.793419 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:14:59.793387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" event={"ID":"56933010-65d6-45d1-8709-c2c2207b1c98","Type":"ContainerStarted","Data":"103d1ff9a6676885b8579ed22a2709844499b8df55f459b0c1549a5baa94e26a"}
Jul 06 21:15:00.800127 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:00.800086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" event={"ID":"56933010-65d6-45d1-8709-c2c2207b1c98","Type":"ContainerStarted","Data":"c8fb6a73244f270b1524f497870225cc61bd2fb9219ab0414913b856d9ee0db7"}
Jul 06 21:15:00.827707 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:00.827648    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podStartSLOduration=4.827633197 podStartE2EDuration="4.827633197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:00.827367003 +0000 UTC m=+1543.287128896" watchObservedRunningTime="2026-07-06 21:15:00.827633197 +0000 UTC m=+1543.287395103"
Jul 06 21:15:01.618403 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:01.618366    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:15:01.797033 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:01.796989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:15:06.213165 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:06.213116    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:15:09.618251 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:09.618213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:15:09.619966 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:09.619936    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:15:16.213119 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:16.213070    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:15:19.618282 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:19.618234    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:15:26.213527 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:26.213478    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:15:29.617921 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:29.617865    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:15:36.213713 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:36.213657    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:15:39.618095 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:39.618039    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:15:46.213464 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:46.213408    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:15:49.618534 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:49.618490    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:15:56.213545 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:56.213494    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:15:59.618290 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:15:59.618226    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:16:06.214080 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:06.214017    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:16:09.618090 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:09.617984    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:16:16.213443 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:16.213399    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:16:19.618669 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:19.618622    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:16:26.213321 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:26.213268    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:16:29.618576 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:29.618520    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:16:36.213512 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:36.213467    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 06 21:16:39.618487 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:39.618428    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:16:46.223519 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:46.223475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:16:46.237038 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:46.237003    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:16:49.618286 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:49.618237    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:16:54.208939 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:54.208890    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"]
Jul 06 21:16:54.235113 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:54.235013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:16:55.600606 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:55.600568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:16:59.618121 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:16:59.618059    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:17:09.240197 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:09.240150    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" containerID="cri-o://10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67" gracePeriod=45
Jul 06 21:17:09.618654 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:09.618605    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:17:10.230116 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.230092    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:17:10.357020 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.356918    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-model-cache\") pod \"343f8755-b17f-44d5-bba1-815638519304\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") "
Jul 06 21:17:10.357020 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-tmp-dir\") pod \"343f8755-b17f-44d5-bba1-815638519304\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") "
Jul 06 21:17:10.357547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbqsl\" (UniqueName: \"kubernetes.io/projected/343f8755-b17f-44d5-bba1-815638519304-kube-api-access-gbqsl\") pod \"343f8755-b17f-44d5-bba1-815638519304\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") "
Jul 06 21:17:10.357547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-kserve-provision-location\") pod \"343f8755-b17f-44d5-bba1-815638519304\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") "
Jul 06 21:17:10.357547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357150    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-home\") pod \"343f8755-b17f-44d5-bba1-815638519304\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") "
Jul 06 21:17:10.357547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-dshm\") pod \"343f8755-b17f-44d5-bba1-815638519304\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") "
Jul 06 21:17:10.357547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343f8755-b17f-44d5-bba1-815638519304-tls-certs\") pod \"343f8755-b17f-44d5-bba1-815638519304\" (UID: \"343f8755-b17f-44d5-bba1-815638519304\") "
Jul 06 21:17:10.357547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357218    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-model-cache" (OuterVolumeSpecName: "model-cache") pod "343f8755-b17f-44d5-bba1-815638519304" (UID: "343f8755-b17f-44d5-bba1-815638519304"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:17:10.357547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357468    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:17:10.357959 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.357857    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-home" (OuterVolumeSpecName: "home") pod "343f8755-b17f-44d5-bba1-815638519304" (UID: "343f8755-b17f-44d5-bba1-815638519304"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:17:10.359522 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.359486    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343f8755-b17f-44d5-bba1-815638519304-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "343f8755-b17f-44d5-bba1-815638519304" (UID: "343f8755-b17f-44d5-bba1-815638519304"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:17:10.359674 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.359647    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-dshm" (OuterVolumeSpecName: "dshm") pod "343f8755-b17f-44d5-bba1-815638519304" (UID: "343f8755-b17f-44d5-bba1-815638519304"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:17:10.359897 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.359770    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343f8755-b17f-44d5-bba1-815638519304-kube-api-access-gbqsl" (OuterVolumeSpecName: "kube-api-access-gbqsl") pod "343f8755-b17f-44d5-bba1-815638519304" (UID: "343f8755-b17f-44d5-bba1-815638519304"). InnerVolumeSpecName "kube-api-access-gbqsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:17:10.369765 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.369727    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "343f8755-b17f-44d5-bba1-815638519304" (UID: "343f8755-b17f-44d5-bba1-815638519304"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:17:10.422735 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.422679    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "343f8755-b17f-44d5-bba1-815638519304" (UID: "343f8755-b17f-44d5-bba1-815638519304"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:17:10.434777 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.434741    2578 generic.go:356] "Generic (PLEG): container finished" podID="343f8755-b17f-44d5-bba1-815638519304" containerID="10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67" exitCode=0
Jul 06 21:17:10.434939 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.434785    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"
Jul 06 21:17:10.434939 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.434789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" event={"ID":"343f8755-b17f-44d5-bba1-815638519304","Type":"ContainerDied","Data":"10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67"}
Jul 06 21:17:10.434939 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.434846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" event={"ID":"343f8755-b17f-44d5-bba1-815638519304","Type":"ContainerDied","Data":"394db80bf7e441928af883bd7591470b6c1dd6c99f6105b51d3a07b158f03cd7"}
Jul 06 21:17:10.434939 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.434877    2578 scope.go:117] "RemoveContainer" containerID="10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67"
Jul 06 21:17:10.444360 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.444337    2578 scope.go:117] "RemoveContainer" containerID="c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129"
Jul 06 21:17:10.452513 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.452488    2578 scope.go:117] "RemoveContainer" containerID="10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67"
Jul 06 21:17:10.452819 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:17:10.452795    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67\": container with ID starting with 10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67 not found: ID does not exist" containerID="10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67"
Jul 06 21:17:10.452865 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.452835    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67"} err="failed to get container status \"10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67\": rpc error: code = NotFound desc = could not find container \"10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67\": container with ID starting with 10afb6ca075f2943958013b231f425c7595d74aec5e641d95a038715b1e18f67 not found: ID does not exist"
Jul 06 21:17:10.452865 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.452860    2578 scope.go:117] "RemoveContainer" containerID="c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129"
Jul 06 21:17:10.453139 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:17:10.453119    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129\": container with ID starting with c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129 not found: ID does not exist" containerID="c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129"
Jul 06 21:17:10.453247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.453144    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129"} err="failed to get container status \"c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129\": rpc error: code = NotFound desc = could not find container \"c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129\": container with ID starting with c9835986cd2de8b089ff301fe5adbbf9136cb414efbc35f202ddbd8723ca3129 not found: ID does not exist"
Jul 06 21:17:10.458287 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.458266    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:17:10.458287 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.458287    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbqsl\" (UniqueName: \"kubernetes.io/projected/343f8755-b17f-44d5-bba1-815638519304-kube-api-access-gbqsl\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:17:10.458429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.458297    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:17:10.458429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.458306    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:17:10.458429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.458314    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343f8755-b17f-44d5-bba1-815638519304-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:17:10.458429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.458321    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343f8755-b17f-44d5-bba1-815638519304-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:17:10.459489 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.459466    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"]
Jul 06 21:17:10.462323 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:10.462300    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc"]
Jul 06 21:17:11.225027 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:11.224981    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Jul 06 21:17:12.159528 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:12.159482    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="343f8755-b17f-44d5-bba1-815638519304" path="/var/lib/kubelet/pods/343f8755-b17f-44d5-bba1-815638519304/volumes"
Jul 06 21:17:19.628008 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:19.627970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:17:19.636075 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:19.636043    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:17:24.035375 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.035340    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"]
Jul 06 21:17:24.035837 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.035772    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main"
Jul 06 21:17:24.035837 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.035790    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main"
Jul 06 21:17:24.035837 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.035800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="storage-initializer"
Jul 06 21:17:24.035837 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.035806    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="storage-initializer"
Jul 06 21:17:24.036006 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.035891    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="343f8755-b17f-44d5-bba1-815638519304" containerName="main"
Jul 06 21:17:24.037914 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.037897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.040984 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.040942    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:17:24.041127 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.041091    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 06 21:17:24.052088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.052054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"]
Jul 06 21:17:24.076130 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.076095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:24.174805 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.174769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.174805 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.174805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.175058 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.174834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-home\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.175058 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.174851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-dshm\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.175058 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.174899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f344c5a1-d9f0-4dad-addc-812662f08e22-tls-certs\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.175058 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.174964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjzn\" (UniqueName: \"kubernetes.io/projected/f344c5a1-d9f0-4dad-addc-812662f08e22-kube-api-access-fgjzn\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.175226 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.175072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-model-cache\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.275907 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.275871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-model-cache\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276099 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.275916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276099 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.275934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276099 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.275960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-home\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276278 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.276094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-dshm\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276278 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.276218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f344c5a1-d9f0-4dad-addc-812662f08e22-tls-certs\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276278 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.276250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjzn\" (UniqueName: \"kubernetes.io/projected/f344c5a1-d9f0-4dad-addc-812662f08e22-kube-api-access-fgjzn\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276497 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.276328    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-model-cache\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276497 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.276352    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-home\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276497 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.276378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.276497 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.276428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.278830 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.278806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-dshm\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.279390 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.279366    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f344c5a1-d9f0-4dad-addc-812662f08e22-tls-certs\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.296016 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.295952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjzn\" (UniqueName: \"kubernetes.io/projected/f344c5a1-d9f0-4dad-addc-812662f08e22-kube-api-access-fgjzn\") pod \"custom-route-timeout-test-kserve-5f665ff7fb-2ltj8\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.319605 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.319569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:24.349686 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.349654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:24.489801 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:24.489704    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"]
Jul 06 21:17:24.492155 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:17:24.492127    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf344c5a1_d9f0_4dad_addc_812662f08e22.slice/crio-bf8a1e178d8342fd1819dfdaef9586b66ea51bd04a463a65041a0c4388d7cd02 WatchSource:0}: Error finding container bf8a1e178d8342fd1819dfdaef9586b66ea51bd04a463a65041a0c4388d7cd02: Status 404 returned error can't find the container with id bf8a1e178d8342fd1819dfdaef9586b66ea51bd04a463a65041a0c4388d7cd02
Jul 06 21:17:25.501917 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:25.501872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" event={"ID":"f344c5a1-d9f0-4dad-addc-812662f08e22","Type":"ContainerStarted","Data":"34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea"}
Jul 06 21:17:25.502416 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:25.501969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" event={"ID":"f344c5a1-d9f0-4dad-addc-812662f08e22","Type":"ContainerStarted","Data":"bf8a1e178d8342fd1819dfdaef9586b66ea51bd04a463a65041a0c4388d7cd02"}
Jul 06 21:17:29.526705 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:29.526666    2578 generic.go:356] "Generic (PLEG): container finished" podID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerID="34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea" exitCode=0
Jul 06 21:17:29.527127 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:29.526747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" event={"ID":"f344c5a1-d9f0-4dad-addc-812662f08e22","Type":"ContainerDied","Data":"34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea"}
Jul 06 21:17:30.535431 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:30.535388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" event={"ID":"f344c5a1-d9f0-4dad-addc-812662f08e22","Type":"ContainerStarted","Data":"6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0"}
Jul 06 21:17:30.559684 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:30.559623    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podStartSLOduration=2.3799368149999998 podStartE2EDuration="6.559603069s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.179666254s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:17:30.557864945 +0000 UTC m=+1693.017626822" watchObservedRunningTime="2026-07-06 21:17:30.559603069 +0000 UTC m=+1693.019364948"
Jul 06 21:17:31.350137 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:31.350082    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:34.350618 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:34.350577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:17:34.352290 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:34.352260    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:17:44.350671 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:44.350578    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:17:49.321173 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:49.321121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:51.028234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:51.028182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:51.058063 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:51.058028    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"]
Jul 06 21:17:51.107453 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:51.107408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:54.350815 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:17:54.350759    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:18:04.350816 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:04.350774    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:18:06.080856 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.080805    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" containerID="cri-o://c8fb6a73244f270b1524f497870225cc61bd2fb9219ab0414913b856d9ee0db7" gracePeriod=45
Jul 06 21:18:06.629200 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.629162    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 06 21:18:06.711963 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.711930    2578 generic.go:356] "Generic (PLEG): container finished" podID="56933010-65d6-45d1-8709-c2c2207b1c98" containerID="c8fb6a73244f270b1524f497870225cc61bd2fb9219ab0414913b856d9ee0db7" exitCode=0
Jul 06 21:18:06.712123 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.712000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" event={"ID":"56933010-65d6-45d1-8709-c2c2207b1c98","Type":"ContainerDied","Data":"c8fb6a73244f270b1524f497870225cc61bd2fb9219ab0414913b856d9ee0db7"}
Jul 06 21:18:06.827323 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.827300    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:18:06.860686 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djbmr\" (UniqueName: \"kubernetes.io/projected/56933010-65d6-45d1-8709-c2c2207b1c98-kube-api-access-djbmr\") pod \"56933010-65d6-45d1-8709-c2c2207b1c98\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") "
Jul 06 21:18:06.860686 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-tmp-dir\") pod \"56933010-65d6-45d1-8709-c2c2207b1c98\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") "
Jul 06 21:18:06.860686 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-model-cache\") pod \"56933010-65d6-45d1-8709-c2c2207b1c98\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") "
Jul 06 21:18:06.860999 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"56933010-65d6-45d1-8709-c2c2207b1c98\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") "
Jul 06 21:18:06.860999 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56933010-65d6-45d1-8709-c2c2207b1c98-tls-certs\") pod \"56933010-65d6-45d1-8709-c2c2207b1c98\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") "
Jul 06 21:18:06.860999 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-home\") pod \"56933010-65d6-45d1-8709-c2c2207b1c98\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") "
Jul 06 21:18:06.860999 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-dshm\") pod \"56933010-65d6-45d1-8709-c2c2207b1c98\" (UID: \"56933010-65d6-45d1-8709-c2c2207b1c98\") "
Jul 06 21:18:06.860999 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.860973    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-model-cache" (OuterVolumeSpecName: "model-cache") pod "56933010-65d6-45d1-8709-c2c2207b1c98" (UID: "56933010-65d6-45d1-8709-c2c2207b1c98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:18:06.861242 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.861226    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:18:06.861913 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.861867    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-home" (OuterVolumeSpecName: "home") pod "56933010-65d6-45d1-8709-c2c2207b1c98" (UID: "56933010-65d6-45d1-8709-c2c2207b1c98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:18:06.863443 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.863409    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56933010-65d6-45d1-8709-c2c2207b1c98-kube-api-access-djbmr" (OuterVolumeSpecName: "kube-api-access-djbmr") pod "56933010-65d6-45d1-8709-c2c2207b1c98" (UID: "56933010-65d6-45d1-8709-c2c2207b1c98"). InnerVolumeSpecName "kube-api-access-djbmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:18:06.863555 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.863446    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-dshm" (OuterVolumeSpecName: "dshm") pod "56933010-65d6-45d1-8709-c2c2207b1c98" (UID: "56933010-65d6-45d1-8709-c2c2207b1c98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:18:06.864349 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.864322    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56933010-65d6-45d1-8709-c2c2207b1c98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56933010-65d6-45d1-8709-c2c2207b1c98" (UID: "56933010-65d6-45d1-8709-c2c2207b1c98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:18:06.865786 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.865763    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e" (OuterVolumeSpecName: "kserve-pvc-source") pod "56933010-65d6-45d1-8709-c2c2207b1c98" (UID: "56933010-65d6-45d1-8709-c2c2207b1c98"). InnerVolumeSpecName "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:18:06.872785 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.872750    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56933010-65d6-45d1-8709-c2c2207b1c98" (UID: "56933010-65d6-45d1-8709-c2c2207b1c98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:18:06.962515 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.962462    2578 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") on node \"ip-10-0-140-38.ec2.internal\" "
Jul 06 21:18:06.962515 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.962502    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56933010-65d6-45d1-8709-c2c2207b1c98-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:18:06.962515 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.962513    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:18:06.962515 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.962521    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:18:06.962946 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.962534    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djbmr\" (UniqueName: \"kubernetes.io/projected/56933010-65d6-45d1-8709-c2c2207b1c98-kube-api-access-djbmr\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:18:06.962946 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.962547    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56933010-65d6-45d1-8709-c2c2207b1c98-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:18:06.990932 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:06.990906    2578 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e") on node "ip-10-0-140-38.ec2.internal"
Jul 06 21:18:07.063461 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:07.063417    2578 reconciler_common.go:299] "Volume detached for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"csi-f383cc3e98112493d8b761a51a111ff1351ce11eb332292421bab9c54484e6b7\""
Jul 06 21:18:07.718030 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:07.717999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck" event={"ID":"56933010-65d6-45d1-8709-c2c2207b1c98","Type":"ContainerDied","Data":"103d1ff9a6676885b8579ed22a2709844499b8df55f459b0c1549a5baa94e26a"}
Jul 06 21:18:07.718030 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:07.718021    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"
Jul 06 21:18:07.718516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:07.718060    2578 scope.go:117] "RemoveContainer" containerID="c8fb6a73244f270b1524f497870225cc61bd2fb9219ab0414913b856d9ee0db7"
Jul 06 21:18:07.742335 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:07.742305    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"]
Jul 06 21:18:07.748732 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:07.748694    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck"]
Jul 06 21:18:08.165563 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:08.165528    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" path="/var/lib/kubelet/pods/56933010-65d6-45d1-8709-c2c2207b1c98/volumes"
Jul 06 21:18:14.351098 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:14.351048    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:18:17.879146 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.879104    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"]
Jul 06 21:18:17.879754 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.879452    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main"
Jul 06 21:18:17.879754 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.879467    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main"
Jul 06 21:18:17.879754 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.879521    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="56933010-65d6-45d1-8709-c2c2207b1c98" containerName="main"
Jul 06 21:18:17.888085 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.888054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:17.891185 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.891157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 06 21:18:17.891382 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.891363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-wtk9s\""
Jul 06 21:18:17.899757 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.899731    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"]
Jul 06 21:18:17.902245 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.902221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"]
Jul 06 21:18:17.902369 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.902321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:17.915863 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.915833    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"]
Jul 06 21:18:17.941194 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.941156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:18:17.960120 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:17.960316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brz4c\" (UniqueName: \"kubernetes.io/projected/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-kube-api-access-brz4c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:17.960316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:17.960316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1954bd7-0063-4163-ad7a-6dc36f860c44-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:17.960316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:17.960316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:17.960580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:17.960580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:17.960580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtq5l\" (UniqueName: \"kubernetes.io/projected/e1954bd7-0063-4163-ad7a-6dc36f860c44-kube-api-access-rtq5l\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:17.960580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:17.960580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:17.960580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:17.960557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.061287 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.061287 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brz4c\" (UniqueName: \"kubernetes.io/projected/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-kube-api-access-brz4c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.061540 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.061540 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1954bd7-0063-4163-ad7a-6dc36f860c44-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.061540 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.061540 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.061540 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.061540 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.061890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtq5l\" (UniqueName: \"kubernetes.io/projected/e1954bd7-0063-4163-ad7a-6dc36f860c44-kube-api-access-rtq5l\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.061890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.061890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.061890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061811    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.061890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.062143 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.061813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.062143 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.062043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.062231 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.062138    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.062231 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.062151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.062231 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.062179    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.063697 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.063677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.064080 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.064055    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.064241 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.064226    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 06 21:18:18.069972 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.069949    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brz4c\" (UniqueName: \"kubernetes.io/projected/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-kube-api-access-brz4c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.070213 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.070193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtq5l\" (UniqueName: \"kubernetes.io/projected/e1954bd7-0063-4163-ad7a-6dc36f860c44-kube-api-access-rtq5l\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.074041 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.074014    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1954bd7-0063-4163-ad7a-6dc36f860c44-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:18.074149 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.074126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:18.166710 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:18.166613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:18:23.513064 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.513022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.523097 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.523067    2578 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.614023 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.613984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.614195 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.614150    2578 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.618406 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.618380    2578 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") DevicePath \"csi-f383cc3e98112493d8b761a51a111ff1351ce11eb332292421bab9c54484e6b7\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.712474 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.712440    2578 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/695b1fedd1efd2a2620b46c540e3c4cbf5ecfc1b083102e8f2c618a1c9ec9ff5/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.734490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.734447    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.816102 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.816067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:23.816291 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.816206    2578 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") DevicePath \"csi-f383cc3e98112493d8b761a51a111ff1351ce11eb332292421bab9c54484e6b7\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:23.820690 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.820663    2578 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") DevicePath \"csi-f383cc3e98112493d8b761a51a111ff1351ce11eb332292421bab9c54484e6b7\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:23.832737 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.832691    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:23.905838 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.905809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-wtk9s\""
Jul 06 21:18:23.912672 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.912646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:23.915948 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:23.915927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:24.090545 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.090452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"]
Jul 06 21:18:24.093763 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:18:24.093734    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1817dcb_ef04_4b91_9ff1_57dc4a7bc46f.slice/crio-180e2f986012e7e3a2a50dcfdf13f1e0db8fdaac3987126a5207ae844f0b2b33 WatchSource:0}: Error finding container 180e2f986012e7e3a2a50dcfdf13f1e0db8fdaac3987126a5207ae844f0b2b33: Status 404 returned error can't find the container with id 180e2f986012e7e3a2a50dcfdf13f1e0db8fdaac3987126a5207ae844f0b2b33
Jul 06 21:18:24.095611 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.095590    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:18:24.110625 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.110590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"]
Jul 06 21:18:24.115176 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:18:24.115142    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1954bd7_0063_4163_ad7a_6dc36f860c44.slice/crio-475cb96f79202e5061d4d6e600d1dab6f2a404c26d3de0358f86d0dfe1ffb5d8 WatchSource:0}: Error finding container 475cb96f79202e5061d4d6e600d1dab6f2a404c26d3de0358f86d0dfe1ffb5d8: Status 404 returned error can't find the container with id 475cb96f79202e5061d4d6e600d1dab6f2a404c26d3de0358f86d0dfe1ffb5d8
Jul 06 21:18:24.350886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.350768    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:18:24.806794 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.806755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" event={"ID":"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f","Type":"ContainerStarted","Data":"180e2f986012e7e3a2a50dcfdf13f1e0db8fdaac3987126a5207ae844f0b2b33"}
Jul 06 21:18:24.808923 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.808887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" event={"ID":"e1954bd7-0063-4163-ad7a-6dc36f860c44","Type":"ContainerStarted","Data":"101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015"}
Jul 06 21:18:24.809086 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.808949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" event={"ID":"e1954bd7-0063-4163-ad7a-6dc36f860c44","Type":"ContainerStarted","Data":"475cb96f79202e5061d4d6e600d1dab6f2a404c26d3de0358f86d0dfe1ffb5d8"}
Jul 06 21:18:24.841904 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:24.841837    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podStartSLOduration=7.841811278 podStartE2EDuration="7.841811278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:18:24.840657606 +0000 UTC m=+1747.300419482" watchObservedRunningTime="2026-07-06 21:18:24.841811278 +0000 UTC m=+1747.301573157"
Jul 06 21:18:25.818490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:25.818448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" event={"ID":"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f","Type":"ContainerStarted","Data":"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"}
Jul 06 21:18:26.827703 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:26.827659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" event={"ID":"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f","Type":"ContainerStarted","Data":"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"}
Jul 06 21:18:26.828231 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:26.827836    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:26.872073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:26.872001    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podStartSLOduration=9.06136645 podStartE2EDuration="9.871976142s" totalImagesPullingTime="810.609692ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:18:26.868024864 +0000 UTC m=+1749.327786764" watchObservedRunningTime="2026-07-06 21:18:26.871976142 +0000 UTC m=+1749.331738020"
Jul 06 21:18:26.896617 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:26.896573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:18:33.913128 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:33.913085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:33.913584 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:33.913170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:33.914685 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:33.914653    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:18:33.916799 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:33.916776    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:33.916906 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:33.916830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:18:33.918155 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:33.918126    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:18:34.350732 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:34.350662    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:18:43.914192 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:43.914134    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:18:43.916684 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:43.916642    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:18:43.928200 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:43.928166    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:18:44.350961 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:44.350904    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:18:53.913634 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:53.913581    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:18:53.917161 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:53.917114    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:18:54.351172 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:18:54.351118    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:19:03.914226 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:03.914167    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:19:03.917173 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:03.917137    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:19:04.350670 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:04.350623    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:19:13.913530 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:13.913419    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:19:13.917168 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:13.917121    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:19:14.350949 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:14.350898    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:19:23.913282 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:23.913227    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:19:23.917196 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:23.917150    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:19:24.351037 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:24.350985    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:19:33.913640 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:33.913523    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:19:33.917176 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:33.917134    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:19:34.361433 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:34.361397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:19:34.369643 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:34.369609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:19:39.640195 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:39.640152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:39.645450 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:39.645417    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"]
Jul 06 21:19:39.737678 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:39.737638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:43.913909 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:43.913856    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:19:43.916413 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:43.916377    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:19:53.913438 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:53.913367    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:19:53.916985 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:53.916954    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:19:54.673384 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:54.673323    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" containerID="cri-o://6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0" gracePeriod=45
Jul 06 21:19:55.362951 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:55.362895    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 06 21:19:55.932263 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:55.932230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:19:56.003516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-model-cache\") pod \"f344c5a1-d9f0-4dad-addc-812662f08e22\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") "
Jul 06 21:19:56.003516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgjzn\" (UniqueName: \"kubernetes.io/projected/f344c5a1-d9f0-4dad-addc-812662f08e22-kube-api-access-fgjzn\") pod \"f344c5a1-d9f0-4dad-addc-812662f08e22\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") "
Jul 06 21:19:56.003516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003504    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f344c5a1-d9f0-4dad-addc-812662f08e22-tls-certs\") pod \"f344c5a1-d9f0-4dad-addc-812662f08e22\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") "
Jul 06 21:19:56.003830 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-kserve-provision-location\") pod \"f344c5a1-d9f0-4dad-addc-812662f08e22\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") "
Jul 06 21:19:56.003830 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-dshm\") pod \"f344c5a1-d9f0-4dad-addc-812662f08e22\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") "
Jul 06 21:19:56.003830 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003614    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-home\") pod \"f344c5a1-d9f0-4dad-addc-812662f08e22\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") "
Jul 06 21:19:56.003830 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-tmp-dir\") pod \"f344c5a1-d9f0-4dad-addc-812662f08e22\" (UID: \"f344c5a1-d9f0-4dad-addc-812662f08e22\") "
Jul 06 21:19:56.003830 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.003732    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-model-cache" (OuterVolumeSpecName: "model-cache") pod "f344c5a1-d9f0-4dad-addc-812662f08e22" (UID: "f344c5a1-d9f0-4dad-addc-812662f08e22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:56.004105 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.004034    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:19:56.004390 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.004356    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-home" (OuterVolumeSpecName: "home") pod "f344c5a1-d9f0-4dad-addc-812662f08e22" (UID: "f344c5a1-d9f0-4dad-addc-812662f08e22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:56.006021 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.005994    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f344c5a1-d9f0-4dad-addc-812662f08e22-kube-api-access-fgjzn" (OuterVolumeSpecName: "kube-api-access-fgjzn") pod "f344c5a1-d9f0-4dad-addc-812662f08e22" (UID: "f344c5a1-d9f0-4dad-addc-812662f08e22"). InnerVolumeSpecName "kube-api-access-fgjzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:19:56.006166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.006079    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f344c5a1-d9f0-4dad-addc-812662f08e22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f344c5a1-d9f0-4dad-addc-812662f08e22" (UID: "f344c5a1-d9f0-4dad-addc-812662f08e22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:19:56.007435 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.007401    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-dshm" (OuterVolumeSpecName: "dshm") pod "f344c5a1-d9f0-4dad-addc-812662f08e22" (UID: "f344c5a1-d9f0-4dad-addc-812662f08e22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:56.022695 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.022621    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f344c5a1-d9f0-4dad-addc-812662f08e22" (UID: "f344c5a1-d9f0-4dad-addc-812662f08e22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:56.072975 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.072937    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f344c5a1-d9f0-4dad-addc-812662f08e22" (UID: "f344c5a1-d9f0-4dad-addc-812662f08e22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:56.105146 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.105112    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjzn\" (UniqueName: \"kubernetes.io/projected/f344c5a1-d9f0-4dad-addc-812662f08e22-kube-api-access-fgjzn\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:19:56.105146 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.105148    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f344c5a1-d9f0-4dad-addc-812662f08e22-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:19:56.105318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.105164    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:19:56.105318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.105176    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:19:56.105318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.105187    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:19:56.105318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.105197    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f344c5a1-d9f0-4dad-addc-812662f08e22-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:19:56.326600 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.326565    2578 generic.go:356] "Generic (PLEG): container finished" podID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerID="6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0" exitCode=0
Jul 06 21:19:56.326803 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.326612    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"
Jul 06 21:19:56.326803 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.326638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" event={"ID":"f344c5a1-d9f0-4dad-addc-812662f08e22","Type":"ContainerDied","Data":"6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0"}
Jul 06 21:19:56.326803 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.326702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8" event={"ID":"f344c5a1-d9f0-4dad-addc-812662f08e22","Type":"ContainerDied","Data":"bf8a1e178d8342fd1819dfdaef9586b66ea51bd04a463a65041a0c4388d7cd02"}
Jul 06 21:19:56.326803 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.326757    2578 scope.go:117] "RemoveContainer" containerID="6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0"
Jul 06 21:19:56.335914 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.335850    2578 scope.go:117] "RemoveContainer" containerID="34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea"
Jul 06 21:19:56.344382 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.344359    2578 scope.go:117] "RemoveContainer" containerID="6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0"
Jul 06 21:19:56.344680 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:19:56.344662    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0\": container with ID starting with 6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0 not found: ID does not exist" containerID="6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0"
Jul 06 21:19:56.344756 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.344693    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0"} err="failed to get container status \"6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0\": rpc error: code = NotFound desc = could not find container \"6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0\": container with ID starting with 6187075ccb3677f30af7371f9f2c24b5a27f8780669fad94f2d8d01ae7daa6c0 not found: ID does not exist"
Jul 06 21:19:56.344825 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.344711    2578 scope.go:117] "RemoveContainer" containerID="34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea"
Jul 06 21:19:56.345089 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:19:56.345071    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea\": container with ID starting with 34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea not found: ID does not exist" containerID="34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea"
Jul 06 21:19:56.345151 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.345094    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea"} err="failed to get container status \"34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea\": rpc error: code = NotFound desc = could not find container \"34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea\": container with ID starting with 34255159a4b05931bd427006d9a199854e333f8853418b368843008a79d526ea not found: ID does not exist"
Jul 06 21:19:56.347903 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.347878    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"]
Jul 06 21:19:56.354449 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:56.354427    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8"]
Jul 06 21:19:58.161691 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:19:58.161656    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" path="/var/lib/kubelet/pods/f344c5a1-d9f0-4dad-addc-812662f08e22/volumes"
Jul 06 21:20:02.075769 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:02.075713    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"]
Jul 06 21:20:02.076170 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:02.075953    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy" containerID="cri-o://15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4" gracePeriod=30
Jul 06 21:20:03.913740 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:03.913667    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:20:03.916616 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:03.916585    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:20:06.171344 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:06.171301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:20:07.335550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.335518    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:20:07.383171 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.383076    2578 generic.go:356] "Generic (PLEG): container finished" podID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerID="15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4" exitCode=0
Jul 06 21:20:07.383171 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.383126    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"
Jul 06 21:20:07.383171 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.383137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" event={"ID":"25b9df5e-4d08-4f3e-996e-0282d903ebeb","Type":"ContainerDied","Data":"15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4"}
Jul 06 21:20:07.383394 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.383209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52" event={"ID":"25b9df5e-4d08-4f3e-996e-0282d903ebeb","Type":"ContainerDied","Data":"1488f4766b6b574ac3b7c7a2e013ecbd29ea52463aa31f0c8e51264449404835"}
Jul 06 21:20:07.383394 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.383243    2578 scope.go:117] "RemoveContainer" containerID="15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4"
Jul 06 21:20:07.391820 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.391796    2578 scope.go:117] "RemoveContainer" containerID="a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3"
Jul 06 21:20:07.401522 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.401497    2578 scope.go:117] "RemoveContainer" containerID="15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4"
Jul 06 21:20:07.401889 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:20:07.401861    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4\": container with ID starting with 15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4 not found: ID does not exist" containerID="15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4"
Jul 06 21:20:07.401987 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.401897    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4"} err="failed to get container status \"15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4\": rpc error: code = NotFound desc = could not find container \"15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4\": container with ID starting with 15c0a725b249e56d02205ae687e08f1f38dd76653faccb8430ecbd87844f35e4 not found: ID does not exist"
Jul 06 21:20:07.401987 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.401917    2578 scope.go:117] "RemoveContainer" containerID="a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3"
Jul 06 21:20:07.402221 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:20:07.402205    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3\": container with ID starting with a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3 not found: ID does not exist" containerID="a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3"
Jul 06 21:20:07.402272 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.402225    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3"} err="failed to get container status \"a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3\": rpc error: code = NotFound desc = could not find container \"a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3\": container with ID starting with a79daff26f313cc98b5237f670b205b116c67d6915928aba65a09c48583fcce3 not found: ID does not exist"
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413217    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-kube-api-access-75m6n\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-envoy\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-token\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-certs\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-podinfo\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-socket\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istiod-ca-cert\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413528    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-credential-socket\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414305 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.413567    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-data\") pod \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\" (UID: \"25b9df5e-4d08-4f3e-996e-0282d903ebeb\") "
Jul 06 21:20:07.414941 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.414441    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:07.414941 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.414457    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:20:07.414941 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.414525    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:07.414941 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.414803    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:07.415156 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.414254    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:07.417276 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.417246    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-kube-api-access-75m6n" (OuterVolumeSpecName: "kube-api-access-75m6n") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "kube-api-access-75m6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:20:07.417421 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.417336    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 21:20:07.418012 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.417985    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:07.418151 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.417986    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "25b9df5e-4d08-4f3e-996e-0282d903ebeb" (UID: "25b9df5e-4d08-4f3e-996e-0282d903ebeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:20:07.515209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515167    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-socket\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515201    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istiod-ca-cert\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515211    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-credential-socket\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515221    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-data\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515572 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515230    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-kube-api-access-75m6n\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515572 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515238    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-envoy\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515572 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515247    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-token\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515572 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515254    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b9df5e-4d08-4f3e-996e-0282d903ebeb-workload-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.515572 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.515262    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b9df5e-4d08-4f3e-996e-0282d903ebeb-istio-podinfo\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:20:07.707088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.707047    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"]
Jul 06 21:20:07.711573 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:07.711541    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52"]
Jul 06 21:20:08.161468 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:08.161429    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" path="/var/lib/kubelet/pods/25b9df5e-4d08-4f3e-996e-0282d903ebeb/volumes"
Jul 06 21:20:13.224329 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224291    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"]
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224632    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="storage-initializer"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224644    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="storage-initializer"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224654    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224661    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224677    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224683    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224690    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224696    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224766    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224775    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f344c5a1-d9f0-4dad-addc-812662f08e22" containerName="main"
Jul 06 21:20:13.224822 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.224783    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b9df5e-4d08-4f3e-996e-0282d903ebeb" containerName="istio-proxy"
Jul 06 21:20:13.229042 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.229016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.231636 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.231611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:20:13.232591 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.232567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 06 21:20:13.249211 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.249179    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"]
Jul 06 21:20:13.257427 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.257392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:20:13.267467 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.267416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-kserve-provision-location\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.267644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.267526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-home\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.267644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.267560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-model-cache\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.267644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.267585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-tmp-dir\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.267644 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.267607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5712bb1a-36b6-44a8-92f9-134506751653-tls-certs\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.267856 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.267694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqkp\" (UniqueName: \"kubernetes.io/projected/5712bb1a-36b6-44a8-92f9-134506751653-kube-api-access-pqqkp\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.267856 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.267752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-dshm\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369219 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-kserve-provision-location\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-home\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-model-cache\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-tmp-dir\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5712bb1a-36b6-44a8-92f9-134506751653-tls-certs\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqkp\" (UniqueName: \"kubernetes.io/projected/5712bb1a-36b6-44a8-92f9-134506751653-kube-api-access-pqqkp\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-dshm\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369798 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369673    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-kserve-provision-location\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369798 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-model-cache\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369906 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369882    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-home\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.369906 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.369882    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-tmp-dir\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.371711 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.371684    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-dshm\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.371933 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.371914    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5712bb1a-36b6-44a8-92f9-134506751653-tls-certs\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.381235 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.381209    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqkp\" (UniqueName: \"kubernetes.io/projected/5712bb1a-36b6-44a8-92f9-134506751653-kube-api-access-pqqkp\") pod \"router-with-refs-test-kserve-69686d4fbd-mlmqd\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.541126 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.541089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:13.690692 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.690652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"]
Jul 06 21:20:13.694877 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:20:13.694842    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5712bb1a_36b6_44a8_92f9_134506751653.slice/crio-bbb59b9f085f5de6909542f2cd03e11cf7a2370fd296b58394074f1d6690e600 WatchSource:0}: Error finding container bbb59b9f085f5de6909542f2cd03e11cf7a2370fd296b58394074f1d6690e600: Status 404 returned error can't find the container with id bbb59b9f085f5de6909542f2cd03e11cf7a2370fd296b58394074f1d6690e600
Jul 06 21:20:13.913460 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.913347    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:20:13.916292 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.916255    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:20:13.916481 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:13.916306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:20:14.019634 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:14.019600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:20:14.417252 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:14.417212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" event={"ID":"5712bb1a-36b6-44a8-92f9-134506751653","Type":"ContainerStarted","Data":"eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5"}
Jul 06 21:20:14.417681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:14.417269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" event={"ID":"5712bb1a-36b6-44a8-92f9-134506751653","Type":"ContainerStarted","Data":"bbb59b9f085f5de6909542f2cd03e11cf7a2370fd296b58394074f1d6690e600"}
Jul 06 21:20:18.442340 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:18.442301    2578 generic.go:356] "Generic (PLEG): container finished" podID="5712bb1a-36b6-44a8-92f9-134506751653" containerID="eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5" exitCode=0
Jul 06 21:20:18.442845 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:18.442380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" event={"ID":"5712bb1a-36b6-44a8-92f9-134506751653","Type":"ContainerDied","Data":"eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5"}
Jul 06 21:20:19.449742 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:19.449680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" event={"ID":"5712bb1a-36b6-44a8-92f9-134506751653","Type":"ContainerStarted","Data":"8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744"}
Jul 06 21:20:19.480886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:19.480833    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podStartSLOduration=2.518854027 podStartE2EDuration="6.480816226s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.961962199s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:20:19.478428356 +0000 UTC m=+1861.938190271" watchObservedRunningTime="2026-07-06 21:20:19.480816226 +0000 UTC m=+1861.940578101"
Jul 06 21:20:19.541437 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:19.541395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:23.541501 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:23.541429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:20:23.543041 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:23.543008    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:20:23.913655 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:23.913545    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:20:23.917198 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:23.917161    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:20:33.542216 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:33.542155    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:20:33.913226 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:33.913102    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:20:33.917025 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:33.916985    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:20:43.542105 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:43.542003    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:20:43.914145 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:43.913786    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:20:43.916461 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:43.916425    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:20:53.542135 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:53.542080    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:20:53.913604 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:53.913499    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:20:53.917357 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:53.917316    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:20:54.422435 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:54.422394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:20:54.524786 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:20:54.524744    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:21:03.542001 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:03.541953    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:21:03.913667 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:03.913525    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 06 21:21:03.917127 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:03.917091    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 06 21:21:13.542147 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:13.542100    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:21:13.923866 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:13.923775    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:21:13.939054 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:13.939009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:21:13.939608 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:13.939574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:21:13.947452 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:13.947420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:21:22.408973 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:22.408931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:21:22.413915 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:22.413886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"]
Jul 06 21:21:22.421965 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:22.421937    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"]
Jul 06 21:21:22.480420 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:22.480383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:21:23.541910 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:23.541866    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:21:33.542029 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:33.541971    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:21:37.438124 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:37.438060    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main" containerID="cri-o://72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d" gracePeriod=45
Jul 06 21:21:37.445676 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:37.445629    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main" containerID="cri-o://101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015" gracePeriod=45
Jul 06 21:21:38.551090 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.550936    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="llm-d-routing-sidecar" containerID="cri-o://7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e" gracePeriod=44
Jul 06 21:21:38.704622 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.704593    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:21:38.792027 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.791984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-home\") pod \"e1954bd7-0063-4163-ad7a-6dc36f860c44\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") "
Jul 06 21:21:38.792027 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.792030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1954bd7-0063-4163-ad7a-6dc36f860c44-tls-certs\") pod \"e1954bd7-0063-4163-ad7a-6dc36f860c44\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") "
Jul 06 21:21:38.792295 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.792062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-dshm\") pod \"e1954bd7-0063-4163-ad7a-6dc36f860c44\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") "
Jul 06 21:21:38.792295 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.792087    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-model-cache\") pod \"e1954bd7-0063-4163-ad7a-6dc36f860c44\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") "
Jul 06 21:21:38.792295 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.792143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtq5l\" (UniqueName: \"kubernetes.io/projected/e1954bd7-0063-4163-ad7a-6dc36f860c44-kube-api-access-rtq5l\") pod \"e1954bd7-0063-4163-ad7a-6dc36f860c44\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") "
Jul 06 21:21:38.792295 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.792174    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-tmp-dir\") pod \"e1954bd7-0063-4163-ad7a-6dc36f860c44\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") "
Jul 06 21:21:38.792295 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.792277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e1954bd7-0063-4163-ad7a-6dc36f860c44\" (UID: \"e1954bd7-0063-4163-ad7a-6dc36f860c44\") "
Jul 06 21:21:38.792677 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.792645    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-home" (OuterVolumeSpecName: "home") pod "e1954bd7-0063-4163-ad7a-6dc36f860c44" (UID: "e1954bd7-0063-4163-ad7a-6dc36f860c44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.794409 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.794380    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1954bd7-0063-4163-ad7a-6dc36f860c44" (UID: "e1954bd7-0063-4163-ad7a-6dc36f860c44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.794946 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.794905    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1954bd7-0063-4163-ad7a-6dc36f860c44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1954bd7-0063-4163-ad7a-6dc36f860c44" (UID: "e1954bd7-0063-4163-ad7a-6dc36f860c44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:21:38.795118 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.795092    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-dshm" (OuterVolumeSpecName: "dshm") pod "e1954bd7-0063-4163-ad7a-6dc36f860c44" (UID: "e1954bd7-0063-4163-ad7a-6dc36f860c44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.795226 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.795111    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1954bd7-0063-4163-ad7a-6dc36f860c44-kube-api-access-rtq5l" (OuterVolumeSpecName: "kube-api-access-rtq5l") pod "e1954bd7-0063-4163-ad7a-6dc36f860c44" (UID: "e1954bd7-0063-4163-ad7a-6dc36f860c44"). InnerVolumeSpecName "kube-api-access-rtq5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:21:38.795989 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.795969    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e" (OuterVolumeSpecName: "kserve-pvc-source") pod "e1954bd7-0063-4163-ad7a-6dc36f860c44" (UID: "e1954bd7-0063-4163-ad7a-6dc36f860c44"). InnerVolumeSpecName "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:21:38.806851 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.806811    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1954bd7-0063-4163-ad7a-6dc36f860c44" (UID: "e1954bd7-0063-4163-ad7a-6dc36f860c44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.811984 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.811958    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:21:38.841665 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.841628    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerID="101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015" exitCode=0
Jul 06 21:21:38.841877 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.841676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"
Jul 06 21:21:38.841877 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.841739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" event={"ID":"e1954bd7-0063-4163-ad7a-6dc36f860c44","Type":"ContainerDied","Data":"101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015"}
Jul 06 21:21:38.841877 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.841797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299" event={"ID":"e1954bd7-0063-4163-ad7a-6dc36f860c44","Type":"ContainerDied","Data":"475cb96f79202e5061d4d6e600d1dab6f2a404c26d3de0358f86d0dfe1ffb5d8"}
Jul 06 21:21:38.841877 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.841823    2578 scope.go:117] "RemoveContainer" containerID="101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015"
Jul 06 21:21:38.843573 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.843543    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerID="72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d" exitCode=0
Jul 06 21:21:38.843681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.843575    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerID="7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e" exitCode=0
Jul 06 21:21:38.843681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.843583    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"
Jul 06 21:21:38.843681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.843628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" event={"ID":"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f","Type":"ContainerDied","Data":"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"}
Jul 06 21:21:38.843864 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.843684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" event={"ID":"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f","Type":"ContainerDied","Data":"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"}
Jul 06 21:21:38.843864 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.843731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9" event={"ID":"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f","Type":"ContainerDied","Data":"180e2f986012e7e3a2a50dcfdf13f1e0db8fdaac3987126a5207ae844f0b2b33"}
Jul 06 21:21:38.853579 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.853551    2578 scope.go:117] "RemoveContainer" containerID="101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015"
Jul 06 21:21:38.853932 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:21:38.853907    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015\": container with ID starting with 101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015 not found: ID does not exist" containerID="101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015"
Jul 06 21:21:38.854041 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.853937    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015"} err="failed to get container status \"101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015\": rpc error: code = NotFound desc = could not find container \"101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015\": container with ID starting with 101abb5bf8112653b948fad704bbf27598e40fbf8cc37c41c0cbb077dd09d015 not found: ID does not exist"
Jul 06 21:21:38.854041 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.853959    2578 scope.go:117] "RemoveContainer" containerID="72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"
Jul 06 21:21:38.864130 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.864086    2578 scope.go:117] "RemoveContainer" containerID="7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"
Jul 06 21:21:38.871868 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.871838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"]
Jul 06 21:21:38.876533 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.876502    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299"]
Jul 06 21:21:38.877245 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.877223    2578 scope.go:117] "RemoveContainer" containerID="72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"
Jul 06 21:21:38.877629 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:21:38.877598    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d\": container with ID starting with 72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d not found: ID does not exist" containerID="72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"
Jul 06 21:21:38.877709 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.877641    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"} err="failed to get container status \"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d\": rpc error: code = NotFound desc = could not find container \"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d\": container with ID starting with 72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d not found: ID does not exist"
Jul 06 21:21:38.877709 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.877668    2578 scope.go:117] "RemoveContainer" containerID="7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"
Jul 06 21:21:38.878001 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:21:38.877982    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e\": container with ID starting with 7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e not found: ID does not exist" containerID="7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"
Jul 06 21:21:38.878066 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.878006    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"} err="failed to get container status \"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e\": rpc error: code = NotFound desc = could not find container \"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e\": container with ID starting with 7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e not found: ID does not exist"
Jul 06 21:21:38.878066 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.878022    2578 scope.go:117] "RemoveContainer" containerID="72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"
Jul 06 21:21:38.878268 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.878252    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d"} err="failed to get container status \"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d\": rpc error: code = NotFound desc = could not find container \"72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d\": container with ID starting with 72ab6ddfdd985c84db03aee7a2327213178e3ad6e45342c9a2c011582b3b841d not found: ID does not exist"
Jul 06 21:21:38.878319 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.878268    2578 scope.go:117] "RemoveContainer" containerID="7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"
Jul 06 21:21:38.878499 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.878456    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e"} err="failed to get container status \"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e\": rpc error: code = NotFound desc = could not find container \"7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e\": container with ID starting with 7df51e9e05d9172b9d8b6a23d944c8737d623174543e99826105e2b67343b78e not found: ID does not exist"
Jul 06 21:21:38.893657 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.893635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") "
Jul 06 21:21:38.893796 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.893769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-model-cache\") pod \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") "
Jul 06 21:21:38.893860 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.893802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-dshm\") pod \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") "
Jul 06 21:21:38.893860 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.893826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tmp-dir\") pod \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") "
Jul 06 21:21:38.893961 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.893856    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-home\") pod \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") "
Jul 06 21:21:38.893961 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.893893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brz4c\" (UniqueName: \"kubernetes.io/projected/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-kube-api-access-brz4c\") pod \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") "
Jul 06 21:21:38.893961 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.893918    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tls-certs\") pod \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\" (UID: \"f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f\") "
Jul 06 21:21:38.894116 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894095    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" (UID: "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.894550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894457    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.894550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894502    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1954bd7-0063-4163-ad7a-6dc36f860c44-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.894550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894519    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.894550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894530    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.894550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894543    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.894890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894545    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-home" (OuterVolumeSpecName: "home") pod "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" (UID: "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.894890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894556    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtq5l\" (UniqueName: \"kubernetes.io/projected/e1954bd7-0063-4163-ad7a-6dc36f860c44-kube-api-access-rtq5l\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.894890 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.894596    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1954bd7-0063-4163-ad7a-6dc36f860c44-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.896454 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.896424    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-kube-api-access-brz4c" (OuterVolumeSpecName: "kube-api-access-brz4c") pod "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" (UID: "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f"). InnerVolumeSpecName "kube-api-access-brz4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:21:38.896550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.896486    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" (UID: "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:21:38.896550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.896495    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-dshm" (OuterVolumeSpecName: "dshm") pod "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" (UID: "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.897451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.897428    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e" (OuterVolumeSpecName: "kserve-pvc-source") pod "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" (UID: "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f"). InnerVolumeSpecName "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:21:38.906687 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.906656    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" (UID: "f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:38.995069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.994979    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.995069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.995013    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brz4c\" (UniqueName: \"kubernetes.io/projected/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-kube-api-access-brz4c\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.995069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.995024    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.995069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.995056    2578 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") on node \"ip-10-0-140-38.ec2.internal\" "
Jul 06 21:21:38.995069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.995070    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:38.995349 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:38.995082    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:21:39.024370 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:39.024346    2578 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e") on node "ip-10-0-140-38.ec2.internal"
Jul 06 21:21:39.095806 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:39.095769    2578 reconciler_common.go:299] "Volume detached for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"csi-f383cc3e98112493d8b761a51a111ff1351ce11eb332292421bab9c54484e6b7\""
Jul 06 21:21:39.175577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:39.175523    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"]
Jul 06 21:21:39.182624 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:39.182590    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9"]
Jul 06 21:21:40.161099 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:40.161064    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" path="/var/lib/kubelet/pods/e1954bd7-0063-4163-ad7a-6dc36f860c44/volumes"
Jul 06 21:21:40.163646 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:40.161466    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" path="/var/lib/kubelet/pods/f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f/volumes"
Jul 06 21:21:43.542148 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:43.542105    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:21:50.503238 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.503192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:21:50.609812 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.609775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:21:50.723609 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.723558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:21:50.764201 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764112    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:21:50.764571 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764547    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main"
Jul 06 21:21:50.764618 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764576    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main"
Jul 06 21:21:50.764618 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764602    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main"
Jul 06 21:21:50.764618 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764612    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main"
Jul 06 21:21:50.764789 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764634    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="llm-d-routing-sidecar"
Jul 06 21:21:50.764789 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764644    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="llm-d-routing-sidecar"
Jul 06 21:21:50.764789 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764755    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="llm-d-routing-sidecar"
Jul 06 21:21:50.764789 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764772    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1954bd7-0063-4163-ad7a-6dc36f860c44" containerName="main"
Jul 06 21:21:50.764789 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.764781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1817dcb-ef04-4b91-9ff1-57dc4a7bc46f" containerName="main"
Jul 06 21:21:50.769302 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.769271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.773423 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.773395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 06 21:21:50.773423 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.773395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bwbs8\""
Jul 06 21:21:50.784780 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.784749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:21:50.794887 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.794853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.795063 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.794904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678ececc-2850-4fbd-8f87-c49d5b100c52-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.795063 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.794994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.795063 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.795040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.795063 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.795060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.795260 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.795146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbsz7\" (UniqueName: \"kubernetes.io/projected/678ececc-2850-4fbd-8f87-c49d5b100c52-kube-api-access-wbsz7\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.834439 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.834398    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:21:50.837770 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.837746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.854832 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.854801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:21:50.879421 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.879357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:21:50.896320 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.896519 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.896519 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.896519 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.896519 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbsz7\" (UniqueName: \"kubernetes.io/projected/678ececc-2850-4fbd-8f87-c49d5b100c52-kube-api-access-wbsz7\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.896802 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896692    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.896802 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896769    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.896940 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.897000 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7z94\" (UniqueName: \"kubernetes.io/projected/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-kube-api-access-k7z94\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.897057 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.896997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.897057 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.897048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.897162 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.897076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.897162 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.897123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678ececc-2850-4fbd-8f87-c49d5b100c52-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.897162 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.897154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.897313 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.897288    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.899045 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.899015    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678ececc-2850-4fbd-8f87-c49d5b100c52-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.899734 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.899693    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678ececc-2850-4fbd-8f87-c49d5b100c52-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.909615 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.909579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbsz7\" (UniqueName: \"kubernetes.io/projected/678ececc-2850-4fbd-8f87-c49d5b100c52-kube-api-access-wbsz7\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"678ececc-2850-4fbd-8f87-c49d5b100c52\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:21:50.998364 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.998577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.998577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7z94\" (UniqueName: \"kubernetes.io/projected/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-kube-api-access-k7z94\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.998577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.998577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.998577 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.998942 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998841    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.998942 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998896    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:50.999037 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:50.998944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:51.000775 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:51.000711    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:51.000886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:51.000832    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:51.011045 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:51.011018    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7z94\" (UniqueName: \"kubernetes.io/projected/793c5d9e-cfef-4986-bbf3-f36fe4151ce3-kube-api-access-k7z94\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"793c5d9e-cfef-4986-bbf3-f36fe4151ce3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:21:53.542428 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:53.542369    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:21:59.882141 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:21:59.882090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:22:03.541732 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:22:03.541688    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:22:13.542008 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:22:13.541909    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:22:23.552596 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:22:23.552558    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:22:23.560587 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:22:23.560561    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:23:03.277865 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:03.277830    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"]
Jul 06 21:23:03.289817 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:03.289776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:03.392198 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:03.392161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:18.301594 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:18.301525    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" containerID="cri-o://8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744" gracePeriod=45
Jul 06 21:23:18.553483 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:18.553384    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 06 21:23:19.057462 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.057435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:23:19.139283 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-dshm\") pod \"5712bb1a-36b6-44a8-92f9-134506751653\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") "
Jul 06 21:23:19.139283 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-model-cache\") pod \"5712bb1a-36b6-44a8-92f9-134506751653\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") "
Jul 06 21:23:19.139283 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-tmp-dir\") pod \"5712bb1a-36b6-44a8-92f9-134506751653\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") "
Jul 06 21:23:19.139283 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5712bb1a-36b6-44a8-92f9-134506751653-tls-certs\") pod \"5712bb1a-36b6-44a8-92f9-134506751653\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") "
Jul 06 21:23:19.139637 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-home\") pod \"5712bb1a-36b6-44a8-92f9-134506751653\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") "
Jul 06 21:23:19.139637 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqqkp\" (UniqueName: \"kubernetes.io/projected/5712bb1a-36b6-44a8-92f9-134506751653-kube-api-access-pqqkp\") pod \"5712bb1a-36b6-44a8-92f9-134506751653\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") "
Jul 06 21:23:19.139637 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-kserve-provision-location\") pod \"5712bb1a-36b6-44a8-92f9-134506751653\" (UID: \"5712bb1a-36b6-44a8-92f9-134506751653\") "
Jul 06 21:23:19.139637 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139486    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-model-cache" (OuterVolumeSpecName: "model-cache") pod "5712bb1a-36b6-44a8-92f9-134506751653" (UID: "5712bb1a-36b6-44a8-92f9-134506751653"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:23:19.139882 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.139742    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:23:19.140224 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.140193    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-home" (OuterVolumeSpecName: "home") pod "5712bb1a-36b6-44a8-92f9-134506751653" (UID: "5712bb1a-36b6-44a8-92f9-134506751653"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:23:19.141563 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.141531    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5712bb1a-36b6-44a8-92f9-134506751653-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5712bb1a-36b6-44a8-92f9-134506751653" (UID: "5712bb1a-36b6-44a8-92f9-134506751653"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:23:19.141682 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.141621    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-dshm" (OuterVolumeSpecName: "dshm") pod "5712bb1a-36b6-44a8-92f9-134506751653" (UID: "5712bb1a-36b6-44a8-92f9-134506751653"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:23:19.141682 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.141636    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5712bb1a-36b6-44a8-92f9-134506751653-kube-api-access-pqqkp" (OuterVolumeSpecName: "kube-api-access-pqqkp") pod "5712bb1a-36b6-44a8-92f9-134506751653" (UID: "5712bb1a-36b6-44a8-92f9-134506751653"). InnerVolumeSpecName "kube-api-access-pqqkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:23:19.155137 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.155102    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5712bb1a-36b6-44a8-92f9-134506751653" (UID: "5712bb1a-36b6-44a8-92f9-134506751653"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:23:19.207780 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.207739    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5712bb1a-36b6-44a8-92f9-134506751653" (UID: "5712bb1a-36b6-44a8-92f9-134506751653"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:23:19.240403 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.240363    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:23:19.240403 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.240398    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:23:19.240403 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.240411    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:23:19.240642 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.240424    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5712bb1a-36b6-44a8-92f9-134506751653-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:23:19.240642 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.240437    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5712bb1a-36b6-44a8-92f9-134506751653-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:23:19.240642 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.240448    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqqkp\" (UniqueName: \"kubernetes.io/projected/5712bb1a-36b6-44a8-92f9-134506751653-kube-api-access-pqqkp\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:23:19.277207 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.277177    2578 generic.go:356] "Generic (PLEG): container finished" podID="5712bb1a-36b6-44a8-92f9-134506751653" containerID="8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744" exitCode=0
Jul 06 21:23:19.277377 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.277215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" event={"ID":"5712bb1a-36b6-44a8-92f9-134506751653","Type":"ContainerDied","Data":"8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744"}
Jul 06 21:23:19.277377 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.277242    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"
Jul 06 21:23:19.277377 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.277256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd" event={"ID":"5712bb1a-36b6-44a8-92f9-134506751653","Type":"ContainerDied","Data":"bbb59b9f085f5de6909542f2cd03e11cf7a2370fd296b58394074f1d6690e600"}
Jul 06 21:23:19.277377 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.277277    2578 scope.go:117] "RemoveContainer" containerID="8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744"
Jul 06 21:23:19.286156 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.286137    2578 scope.go:117] "RemoveContainer" containerID="eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5"
Jul 06 21:23:19.294117 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.294095    2578 scope.go:117] "RemoveContainer" containerID="8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744"
Jul 06 21:23:19.294372 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:23:19.294351    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744\": container with ID starting with 8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744 not found: ID does not exist" containerID="8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744"
Jul 06 21:23:19.294472 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.294383    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744"} err="failed to get container status \"8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744\": rpc error: code = NotFound desc = could not find container \"8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744\": container with ID starting with 8cb3c58214fd88a788f5a5636081fe763f3cc78fedb478fc8c712169b0364744 not found: ID does not exist"
Jul 06 21:23:19.294472 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.294407    2578 scope.go:117] "RemoveContainer" containerID="eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5"
Jul 06 21:23:19.294644 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:23:19.294629    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5\": container with ID starting with eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5 not found: ID does not exist" containerID="eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5"
Jul 06 21:23:19.294694 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.294650    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5"} err="failed to get container status \"eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5\": rpc error: code = NotFound desc = could not find container \"eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5\": container with ID starting with eaf747c4731231bd68d0028d7739b255e8787da9900d83fbd00e21caae5816d5 not found: ID does not exist"
Jul 06 21:23:19.301755 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.301733    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"]
Jul 06 21:23:19.309087 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:19.309062    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd"]
Jul 06 21:23:20.159980 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:20.159948    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5712bb1a-36b6-44a8-92f9-134506751653" path="/var/lib/kubelet/pods/5712bb1a-36b6-44a8-92f9-134506751653/volumes"
Jul 06 21:23:39.663474 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.663381    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"]
Jul 06 21:23:39.664088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.663776    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main"
Jul 06 21:23:39.664088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.663794    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main"
Jul 06 21:23:39.664088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.663814    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="storage-initializer"
Jul 06 21:23:39.664088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.663823    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="storage-initializer"
Jul 06 21:23:39.664088 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.663898    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5712bb1a-36b6-44a8-92f9-134506751653" containerName="main"
Jul 06 21:23:39.666968 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.666946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.673005 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.672974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:23:39.673179 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.673028    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 06 21:23:39.673319 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.673299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-dl444\""
Jul 06 21:23:39.684274 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.684246    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"]
Jul 06 21:23:39.687761 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.687739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"]
Jul 06 21:23:39.687880 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.687805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.707848 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.707817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"]
Jul 06 21:23:39.726208 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.726172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:39.832908 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.832871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.833090 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.832916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.833090 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.832953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.833090 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.832968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.833090 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.833090 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.833325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.833325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515935e9-dfd1-4925-8d19-7b48a88ac125-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.833325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ptq\" (UniqueName: \"kubernetes.io/projected/515935e9-dfd1-4925-8d19-7b48a88ac125-kube-api-access-p9ptq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.833325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.833325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.833325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.833580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.833580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.833407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbvg4\" (UniqueName: \"kubernetes.io/projected/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kube-api-access-bbvg4\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.934481 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.934481 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbvg4\" (UniqueName: \"kubernetes.io/projected/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kube-api-access-bbvg4\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.934481 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.934839 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.934839 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.934839 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.934839 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.934839 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.934839 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515935e9-dfd1-4925-8d19-7b48a88ac125-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ptq\" (UniqueName: \"kubernetes.io/projected/515935e9-dfd1-4925-8d19-7b48a88ac125-kube-api-access-p9ptq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934890    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934894    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934986    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.934970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.935051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.935063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.935163 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.935138    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935619 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.935342    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.935619 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.935367    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.937217 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.937189    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.937381 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.937357    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.937464 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.937448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.937764 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.937746    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515935e9-dfd1-4925-8d19-7b48a88ac125-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.948706 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.948680    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbvg4\" (UniqueName: \"kubernetes.io/projected/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kube-api-access-bbvg4\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:39.948944 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.948923    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ptq\" (UniqueName: \"kubernetes.io/projected/515935e9-dfd1-4925-8d19-7b48a88ac125-kube-api-access-p9ptq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.976923 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.976885    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:39.998367 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:39.998338    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:40.049337 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.049296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:40.134931 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.134896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"]
Jul 06 21:23:40.138436 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:23:40.138403    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515935e9_dfd1_4925_8d19_7b48a88ac125.slice/crio-8c77c06978b6572e56f01e9245f937efa8d0f17ce3b7ada7901686c59bfa6e87 WatchSource:0}: Error finding container 8c77c06978b6572e56f01e9245f937efa8d0f17ce3b7ada7901686c59bfa6e87: Status 404 returned error can't find the container with id 8c77c06978b6572e56f01e9245f937efa8d0f17ce3b7ada7901686c59bfa6e87
Jul 06 21:23:40.160770 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.160752    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"]
Jul 06 21:23:40.167752 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:23:40.167705    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f70b6a_d75d_4b03_b0bd_e8f56081d326.slice/crio-f03643657221be66b0a338da71a19fd1ac3f05fcdef45ff018956221cf7eabee WatchSource:0}: Error finding container f03643657221be66b0a338da71a19fd1ac3f05fcdef45ff018956221cf7eabee: Status 404 returned error can't find the container with id f03643657221be66b0a338da71a19fd1ac3f05fcdef45ff018956221cf7eabee
Jul 06 21:23:40.366919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.366881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerStarted","Data":"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"}
Jul 06 21:23:40.367109 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.366947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerStarted","Data":"8c77c06978b6572e56f01e9245f937efa8d0f17ce3b7ada7901686c59bfa6e87"}
Jul 06 21:23:40.367109 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.366979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:40.368581 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.368558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" event={"ID":"69f70b6a-d75d-4b03-b0bd-e8f56081d326","Type":"ContainerStarted","Data":"7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689"}
Jul 06 21:23:40.368685 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:40.368601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" event={"ID":"69f70b6a-d75d-4b03-b0bd-e8f56081d326","Type":"ContainerStarted","Data":"f03643657221be66b0a338da71a19fd1ac3f05fcdef45ff018956221cf7eabee"}
Jul 06 21:23:41.384613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:41.384467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerStarted","Data":"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"}
Jul 06 21:23:44.406665 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:44.406625    2578 generic.go:356] "Generic (PLEG): container finished" podID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerID="7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689" exitCode=0
Jul 06 21:23:44.407402 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:44.406705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" event={"ID":"69f70b6a-d75d-4b03-b0bd-e8f56081d326","Type":"ContainerDied","Data":"7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689"}
Jul 06 21:23:44.408778 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:44.408752    2578 generic.go:356] "Generic (PLEG): container finished" podID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerID="c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac" exitCode=0
Jul 06 21:23:44.408877 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:44.408834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerDied","Data":"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"}
Jul 06 21:23:45.417769 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:45.417704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" event={"ID":"69f70b6a-d75d-4b03-b0bd-e8f56081d326","Type":"ContainerStarted","Data":"72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05"}
Jul 06 21:23:45.420439 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:45.420411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerStarted","Data":"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"}
Jul 06 21:23:45.447782 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:45.447697    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podStartSLOduration=2.471964752 podStartE2EDuration="6.447675092s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.97571034s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:23:45.444225423 +0000 UTC m=+2067.903987298" watchObservedRunningTime="2026-07-06 21:23:45.447675092 +0000 UTC m=+2067.907436971"
Jul 06 21:23:45.474762 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:45.474685    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podStartSLOduration=3.025567794 podStartE2EDuration="6.474669187s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.449101393s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:23:45.472979417 +0000 UTC m=+2067.932741330" watchObservedRunningTime="2026-07-06 21:23:45.474669187 +0000 UTC m=+2067.934431064"
Jul 06 21:23:47.437069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:47.437004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:47.500313 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:47.500274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:49.977970 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:49.977927    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:49.978516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:49.977999    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:49.979574 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:49.979536    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:23:49.999324 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:49.999278    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:49.999501 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:49.999358    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:23:50.000843 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:50.000811    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:23:53.781062 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:23:53.780988    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:23:53.848231 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:23:53.848171    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:23:54.463833 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:54.463799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:23:54.463833 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:54.463817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:23:59.978169 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:59.978118    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:23:59.995004 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:59.994975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:23:59.999756 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:23:59.999700    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:24:09.977917 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:09.977862    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:24:09.999071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:09.999020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:24:10.842561 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:10.842518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:24:19.977733 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:19.977666    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:24:19.999164 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:19.999114    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:24:29.977838 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:29.977784    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:24:29.999472 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:29.999427    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:24:39.978251 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:39.978197    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:24:39.999069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:39.999020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:24:49.977410 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:49.977353    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:24:49.999355 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:49.999315    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:24:59.978206 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:59.978146    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:24:59.999513 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:24:59.999473    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:25:09.977456 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:09.977325    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:25:09.998959 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:09.998920    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:25:19.978073 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:19.978020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:25:19.999255 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:19.999201    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:25:29.977772 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:29.977707    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:25:29.999676 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:29.999636    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:25:39.977920 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:39.977869    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:25:39.999572 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:39.999530    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:25:49.978209 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:49.978148    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:25:49.999630 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:49.999588    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:25:53.232298 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:53.232259    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:25:57.464865 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:25:57.464815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:25:57.464865 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:25:57.464815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:25:58.072281 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:58.072250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:25:58.072498 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:58.072250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:25:59.977481 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:59.977427    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:25:59.999102 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:25:59.999063    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:26:09.977582 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:09.977536    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:26:09.999618 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:09.999572    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:26:19.978437 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:19.978391    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 06 21:26:19.999424 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:19.999388    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 06 21:26:29.987490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:29.987456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:26:29.999828 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:29.999801    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:26:30.009537 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:30.009508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:26:30.017309 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:26:30.017288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:27:03.073399 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:03.073366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:27:03.125863 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:03.125824    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"]
Jul 06 21:27:03.127524 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:03.127500    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"]
Jul 06 21:27:04.211800 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:04.211760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:27:18.149137 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:18.149070    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main" containerID="cri-o://72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05" gracePeriod=45
Jul 06 21:27:18.150105 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:18.150051    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main" containerID="cri-o://6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9" gracePeriod=45
Jul 06 21:27:18.961452 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:18.961415    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="llm-d-routing-sidecar" containerID="cri-o://94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf" gracePeriod=45
Jul 06 21:27:19.099571 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.099541    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:27:19.203345 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.203320    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:27:19.296971 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.296932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbvg4\" (UniqueName: \"kubernetes.io/projected/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kube-api-access-bbvg4\") pod \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") "
Jul 06 21:27:19.297166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.296991    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tls-certs\") pod \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") "
Jul 06 21:27:19.297166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.297024    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-dshm\") pod \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") "
Jul 06 21:27:19.297166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.297052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kserve-provision-location\") pod \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") "
Jul 06 21:27:19.297166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.297094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-model-cache\") pod \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") "
Jul 06 21:27:19.297166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.297119    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-home\") pod \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") "
Jul 06 21:27:19.297166 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.297162    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tmp-dir\") pod \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\" (UID: \"69f70b6a-d75d-4b03-b0bd-e8f56081d326\") "
Jul 06 21:27:19.297472 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.297413    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-model-cache" (OuterVolumeSpecName: "model-cache") pod "69f70b6a-d75d-4b03-b0bd-e8f56081d326" (UID: "69f70b6a-d75d-4b03-b0bd-e8f56081d326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.297893 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.297857    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-home" (OuterVolumeSpecName: "home") pod "69f70b6a-d75d-4b03-b0bd-e8f56081d326" (UID: "69f70b6a-d75d-4b03-b0bd-e8f56081d326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.299610 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.299546    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69f70b6a-d75d-4b03-b0bd-e8f56081d326" (UID: "69f70b6a-d75d-4b03-b0bd-e8f56081d326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:27:19.299753 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.299655    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-dshm" (OuterVolumeSpecName: "dshm") pod "69f70b6a-d75d-4b03-b0bd-e8f56081d326" (UID: "69f70b6a-d75d-4b03-b0bd-e8f56081d326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.299753 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.299673    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kube-api-access-bbvg4" (OuterVolumeSpecName: "kube-api-access-bbvg4") pod "69f70b6a-d75d-4b03-b0bd-e8f56081d326" (UID: "69f70b6a-d75d-4b03-b0bd-e8f56081d326"). InnerVolumeSpecName "kube-api-access-bbvg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:27:19.309802 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.309774    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69f70b6a-d75d-4b03-b0bd-e8f56081d326" (UID: "69f70b6a-d75d-4b03-b0bd-e8f56081d326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.361578 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.361543    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69f70b6a-d75d-4b03-b0bd-e8f56081d326" (UID: "69f70b6a-d75d-4b03-b0bd-e8f56081d326"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.397836 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.397802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-tmp-dir\") pod \"515935e9-dfd1-4925-8d19-7b48a88ac125\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") "
Jul 06 21:27:19.398012 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.397854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515935e9-dfd1-4925-8d19-7b48a88ac125-tls-certs\") pod \"515935e9-dfd1-4925-8d19-7b48a88ac125\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") "
Jul 06 21:27:19.398012 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.397900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-dshm\") pod \"515935e9-dfd1-4925-8d19-7b48a88ac125\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") "
Jul 06 21:27:19.398012 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.397929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-home\") pod \"515935e9-dfd1-4925-8d19-7b48a88ac125\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") "
Jul 06 21:27:19.398185 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398061    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-model-cache\") pod \"515935e9-dfd1-4925-8d19-7b48a88ac125\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") "
Jul 06 21:27:19.398185 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-kserve-provision-location\") pod \"515935e9-dfd1-4925-8d19-7b48a88ac125\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") "
Jul 06 21:27:19.398185 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9ptq\" (UniqueName: \"kubernetes.io/projected/515935e9-dfd1-4925-8d19-7b48a88ac125-kube-api-access-p9ptq\") pod \"515935e9-dfd1-4925-8d19-7b48a88ac125\" (UID: \"515935e9-dfd1-4925-8d19-7b48a88ac125\") "
Jul 06 21:27:19.398387 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398355    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-model-cache" (OuterVolumeSpecName: "model-cache") pod "515935e9-dfd1-4925-8d19-7b48a88ac125" (UID: "515935e9-dfd1-4925-8d19-7b48a88ac125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.398628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398546    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbvg4\" (UniqueName: \"kubernetes.io/projected/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kube-api-access-bbvg4\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.398628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398572    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.398628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398590    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.398628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398603    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.398628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398615    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.398628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398628    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.398628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398639    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.399083 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398652    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f70b6a-d75d-4b03-b0bd-e8f56081d326-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.399083 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.398689    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-home" (OuterVolumeSpecName: "home") pod "515935e9-dfd1-4925-8d19-7b48a88ac125" (UID: "515935e9-dfd1-4925-8d19-7b48a88ac125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.400135 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.400108    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515935e9-dfd1-4925-8d19-7b48a88ac125-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "515935e9-dfd1-4925-8d19-7b48a88ac125" (UID: "515935e9-dfd1-4925-8d19-7b48a88ac125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:27:19.400325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.400302    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515935e9-dfd1-4925-8d19-7b48a88ac125-kube-api-access-p9ptq" (OuterVolumeSpecName: "kube-api-access-p9ptq") pod "515935e9-dfd1-4925-8d19-7b48a88ac125" (UID: "515935e9-dfd1-4925-8d19-7b48a88ac125"). InnerVolumeSpecName "kube-api-access-p9ptq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:27:19.400447 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.400426    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-dshm" (OuterVolumeSpecName: "dshm") pod "515935e9-dfd1-4925-8d19-7b48a88ac125" (UID: "515935e9-dfd1-4925-8d19-7b48a88ac125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.410466 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.410435    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "515935e9-dfd1-4925-8d19-7b48a88ac125" (UID: "515935e9-dfd1-4925-8d19-7b48a88ac125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.429742 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.429695    2578 generic.go:356] "Generic (PLEG): container finished" podID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerID="72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05" exitCode=0
Jul 06 21:27:19.429903 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.429749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" event={"ID":"69f70b6a-d75d-4b03-b0bd-e8f56081d326","Type":"ContainerDied","Data":"72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05"}
Jul 06 21:27:19.429903 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.429776    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"
Jul 06 21:27:19.429903 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.429810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv" event={"ID":"69f70b6a-d75d-4b03-b0bd-e8f56081d326","Type":"ContainerDied","Data":"f03643657221be66b0a338da71a19fd1ac3f05fcdef45ff018956221cf7eabee"}
Jul 06 21:27:19.429903 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.429840    2578 scope.go:117] "RemoveContainer" containerID="72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05"
Jul 06 21:27:19.431785 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.431759    2578 generic.go:356] "Generic (PLEG): container finished" podID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerID="6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9" exitCode=0
Jul 06 21:27:19.431785 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.431781    2578 generic.go:356] "Generic (PLEG): container finished" podID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerID="94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf" exitCode=0
Jul 06 21:27:19.431970 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.431836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"
Jul 06 21:27:19.431970 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.431837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerDied","Data":"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"}
Jul 06 21:27:19.431970 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.431955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerDied","Data":"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"}
Jul 06 21:27:19.432133 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.431987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx" event={"ID":"515935e9-dfd1-4925-8d19-7b48a88ac125","Type":"ContainerDied","Data":"8c77c06978b6572e56f01e9245f937efa8d0f17ce3b7ada7901686c59bfa6e87"}
Jul 06 21:27:19.439793 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.439770    2578 scope.go:117] "RemoveContainer" containerID="7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689"
Jul 06 21:27:19.447697 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.447680    2578 scope.go:117] "RemoveContainer" containerID="72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05"
Jul 06 21:27:19.447999 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:27:19.447981    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05\": container with ID starting with 72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05 not found: ID does not exist" containerID="72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05"
Jul 06 21:27:19.448066 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.448008    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05"} err="failed to get container status \"72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05\": rpc error: code = NotFound desc = could not find container \"72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05\": container with ID starting with 72963cd0dd4c209cef40c1a7b6e0793e46d82c9fad0a4632d16bfd746a1cab05 not found: ID does not exist"
Jul 06 21:27:19.448066 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.448024    2578 scope.go:117] "RemoveContainer" containerID="7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689"
Jul 06 21:27:19.448275 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:27:19.448254    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689\": container with ID starting with 7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689 not found: ID does not exist" containerID="7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689"
Jul 06 21:27:19.448428 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.448285    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689"} err="failed to get container status \"7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689\": rpc error: code = NotFound desc = could not find container \"7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689\": container with ID starting with 7333f1928ea39910e762184af2414ebdbdb0d262ef9697edf5c94c76317a4689 not found: ID does not exist"
Jul 06 21:27:19.448428 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.448308    2578 scope.go:117] "RemoveContainer" containerID="6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"
Jul 06 21:27:19.453601 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.453580    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"]
Jul 06 21:27:19.457249 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.457230    2578 scope.go:117] "RemoveContainer" containerID="c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"
Jul 06 21:27:19.459572 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.459548    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv"]
Jul 06 21:27:19.465316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.465297    2578 scope.go:117] "RemoveContainer" containerID="94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"
Jul 06 21:27:19.473668 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.473650    2578 scope.go:117] "RemoveContainer" containerID="6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"
Jul 06 21:27:19.473957 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:27:19.473938    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9\": container with ID starting with 6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9 not found: ID does not exist" containerID="6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"
Jul 06 21:27:19.474034 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.473963    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"} err="failed to get container status \"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9\": rpc error: code = NotFound desc = could not find container \"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9\": container with ID starting with 6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9 not found: ID does not exist"
Jul 06 21:27:19.474034 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.473981    2578 scope.go:117] "RemoveContainer" containerID="c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"
Jul 06 21:27:19.474209 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:27:19.474193    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac\": container with ID starting with c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac not found: ID does not exist" containerID="c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"
Jul 06 21:27:19.474262 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474213    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"} err="failed to get container status \"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac\": rpc error: code = NotFound desc = could not find container \"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac\": container with ID starting with c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac not found: ID does not exist"
Jul 06 21:27:19.474262 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474227    2578 scope.go:117] "RemoveContainer" containerID="94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"
Jul 06 21:27:19.474502 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:27:19.474473    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf\": container with ID starting with 94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf not found: ID does not exist" containerID="94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"
Jul 06 21:27:19.474549 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474507    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"} err="failed to get container status \"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf\": rpc error: code = NotFound desc = could not find container \"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf\": container with ID starting with 94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf not found: ID does not exist"
Jul 06 21:27:19.474549 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474525    2578 scope.go:117] "RemoveContainer" containerID="6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"
Jul 06 21:27:19.474629 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474548    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "515935e9-dfd1-4925-8d19-7b48a88ac125" (UID: "515935e9-dfd1-4925-8d19-7b48a88ac125"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:27:19.474783 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474763    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9"} err="failed to get container status \"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9\": rpc error: code = NotFound desc = could not find container \"6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9\": container with ID starting with 6ceb7c9ebe2bf0f68b3d91898c8566f89319f1d907ef7640952cfacd4a36d0b9 not found: ID does not exist"
Jul 06 21:27:19.474835 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474784    2578 scope.go:117] "RemoveContainer" containerID="c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"
Jul 06 21:27:19.475002 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.474983    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac"} err="failed to get container status \"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac\": rpc error: code = NotFound desc = could not find container \"c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac\": container with ID starting with c45709f3313fbdebdb44bc07a7d8d59ba1df2187425965c6f42590c333015eac not found: ID does not exist"
Jul 06 21:27:19.475047 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.475003    2578 scope.go:117] "RemoveContainer" containerID="94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"
Jul 06 21:27:19.475198 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.475182    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf"} err="failed to get container status \"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf\": rpc error: code = NotFound desc = could not find container \"94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf\": container with ID starting with 94c4d2b11c0c674b625960894694cdb9d37b20c6be95e595c58be63c0b30aecf not found: ID does not exist"
Jul 06 21:27:19.499731 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.499692    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.499731 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.499734    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515935e9-dfd1-4925-8d19-7b48a88ac125-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.499915 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.499745    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.499915 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.499755    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.499915 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.499763    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515935e9-dfd1-4925-8d19-7b48a88ac125-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.499915 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.499771    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ptq\" (UniqueName: \"kubernetes.io/projected/515935e9-dfd1-4925-8d19-7b48a88ac125-kube-api-access-p9ptq\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:27:19.779740 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.779687    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"]
Jul 06 21:27:19.794568 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:19.794535    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx"]
Jul 06 21:27:20.159831 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:20.159801    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" path="/var/lib/kubelet/pods/515935e9-dfd1-4925-8d19-7b48a88ac125/volumes"
Jul 06 21:27:20.160282 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:20.160269    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" path="/var/lib/kubelet/pods/69f70b6a-d75d-4b03-b0bd-e8f56081d326/volumes"
Jul 06 21:27:34.768858 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.768820    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"]
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769130    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="llm-d-routing-sidecar"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769142    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="llm-d-routing-sidecar"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769150    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="storage-initializer"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769156    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="storage-initializer"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769163    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769169    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769176    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769182    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769189    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="storage-initializer"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="storage-initializer"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769240    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="main"
Jul 06 21:27:34.769247 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769250    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="515935e9-dfd1-4925-8d19-7b48a88ac125" containerName="llm-d-routing-sidecar"
Jul 06 21:27:34.769618 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.769256    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f70b6a-d75d-4b03-b0bd-e8f56081d326" containerName="main"
Jul 06 21:27:34.772051 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.772034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.775008 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.774986    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-6k8mb\""
Jul 06 21:27:34.775164 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.774994    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:27:34.775164 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.775085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 06 21:27:34.790265 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.790232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"]
Jul 06 21:27:34.802520 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.802488    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"]
Jul 06 21:27:34.805457 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.805440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.821742 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.821696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"]
Jul 06 21:27:34.838080 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.838251 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-dshm\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.838251 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.838251 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4gv\" (UniqueName: \"kubernetes.io/projected/b61ec122-85b1-41b5-874e-9f1f11e5d956-kube-api-access-vg4gv\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.838251 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-home\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.838404 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.838404 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.838404 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.838404 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmn5g\" (UniqueName: \"kubernetes.io/projected/da56c2c4-c212-413e-99a1-81708d53196b-kube-api-access-kmn5g\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.838550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.838550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.838550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec122-85b1-41b5-874e-9f1f11e5d956-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.838550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.838550 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.838522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da56c2c4-c212-413e-99a1-81708d53196b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.939981 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.939944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.939981 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.939992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da56c2c4-c212-413e-99a1-81708d53196b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940211 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940211 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-dshm\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940211 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940369 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4gv\" (UniqueName: \"kubernetes.io/projected/b61ec122-85b1-41b5-874e-9f1f11e5d956-kube-api-access-vg4gv\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940369 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-home\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940369 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940523 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940523 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940405    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940523 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940426    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940523 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940523 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940472    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940523 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmn5g\" (UniqueName: \"kubernetes.io/projected/da56c2c4-c212-413e-99a1-81708d53196b-kube-api-access-kmn5g\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940909 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940909 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940909 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec122-85b1-41b5-874e-9f1f11e5d956-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940909 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940661    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-home\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.940909 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.940909 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.940890    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.941194 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.941015    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.941194 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.941032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.942479 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.942448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-dshm\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.942845 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.942826    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da56c2c4-c212-413e-99a1-81708d53196b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.943024 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.943008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.943388 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.943367    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec122-85b1-41b5-874e-9f1f11e5d956-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.948817 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.948768    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4gv\" (UniqueName: \"kubernetes.io/projected/b61ec122-85b1-41b5-874e-9f1f11e5d956-kube-api-access-vg4gv\") pod \"custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:34.948986 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.948969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmn5g\" (UniqueName: \"kubernetes.io/projected/da56c2c4-c212-413e-99a1-81708d53196b-kube-api-access-kmn5g\") pod \"custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:34.992640 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:34.992606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:27:35.081134 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.081094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:35.115087 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.115046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:35.245108 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.245076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"]
Jul 06 21:27:35.245298 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:27:35.245277    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61ec122_85b1_41b5_874e_9f1f11e5d956.slice/crio-534d692480a220cdfbab0b86b1a8163053c768046c5294c8fa1fb887b4fcf5a5 WatchSource:0}: Error finding container 534d692480a220cdfbab0b86b1a8163053c768046c5294c8fa1fb887b4fcf5a5: Status 404 returned error can't find the container with id 534d692480a220cdfbab0b86b1a8163053c768046c5294c8fa1fb887b4fcf5a5
Jul 06 21:27:35.267068 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.266992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"]
Jul 06 21:27:35.270272 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:27:35.270247    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda56c2c4_c212_413e_99a1_81708d53196b.slice/crio-18d60159d7e7bb3eb5ace0a1a347bfce18eaa5193b03352fa559c52725cfcf1a WatchSource:0}: Error finding container 18d60159d7e7bb3eb5ace0a1a347bfce18eaa5193b03352fa559c52725cfcf1a: Status 404 returned error can't find the container with id 18d60159d7e7bb3eb5ace0a1a347bfce18eaa5193b03352fa559c52725cfcf1a
Jul 06 21:27:35.287192 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.287115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:27:35.503451 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.503338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerStarted","Data":"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"}
Jul 06 21:27:35.503631 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.503447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:35.503631 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.503497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerStarted","Data":"534d692480a220cdfbab0b86b1a8163053c768046c5294c8fa1fb887b4fcf5a5"}
Jul 06 21:27:35.504945 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.504920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" event={"ID":"da56c2c4-c212-413e-99a1-81708d53196b","Type":"ContainerStarted","Data":"fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6"}
Jul 06 21:27:35.505054 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:35.504969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" event={"ID":"da56c2c4-c212-413e-99a1-81708d53196b","Type":"ContainerStarted","Data":"18d60159d7e7bb3eb5ace0a1a347bfce18eaa5193b03352fa559c52725cfcf1a"}
Jul 06 21:27:36.513189 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:36.513139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerStarted","Data":"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"}
Jul 06 21:27:39.528190 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:39.528154    2578 generic.go:356] "Generic (PLEG): container finished" podID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerID="84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e" exitCode=0
Jul 06 21:27:39.528601 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:39.528230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerDied","Data":"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"}
Jul 06 21:27:40.535554 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:40.535516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerStarted","Data":"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"}
Jul 06 21:27:40.564350 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:40.564289    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podStartSLOduration=2.742205279 podStartE2EDuration="6.564263829s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82205855s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:27:40.563121533 +0000 UTC m=+2303.022883423" watchObservedRunningTime="2026-07-06 21:27:40.564263829 +0000 UTC m=+2303.024025706"
Jul 06 21:27:41.541733 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:41.541681    2578 generic.go:356] "Generic (PLEG): container finished" podID="da56c2c4-c212-413e-99a1-81708d53196b" containerID="fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6" exitCode=0
Jul 06 21:27:41.542120 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:41.541766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" event={"ID":"da56c2c4-c212-413e-99a1-81708d53196b","Type":"ContainerDied","Data":"fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6"}
Jul 06 21:27:42.548381 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:42.548342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" event={"ID":"da56c2c4-c212-413e-99a1-81708d53196b","Type":"ContainerStarted","Data":"03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71"}
Jul 06 21:27:42.584562 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:42.584494    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podStartSLOduration=3.055175431 podStartE2EDuration="8.584471612s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.529296181s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:27:42.583992132 +0000 UTC m=+2305.043754011" watchObservedRunningTime="2026-07-06 21:27:42.584471612 +0000 UTC m=+2305.044233489"
Jul 06 21:27:44.649498 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:44.649459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:27:45.081964 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:45.081909    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:45.082180 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:45.081988    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:45.083369 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:45.083329    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:27:45.102156 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:45.102127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:27:45.115893 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:45.115851    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:45.116089 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:45.115934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:27:45.117388 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:45.117352    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:27:46.695360 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:46.695320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:27:55.082132 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:55.082084    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:27:55.115870 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:27:55.115824    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:28:01.073466 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:28:01.073409    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:28:01.074046 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:28:01.073475    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:28:01.321578 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:01.321537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:28:01.427015 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:01.426976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:28:01.641389 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:01.641306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:28:01.641389 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:01.641332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:28:05.082179 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:05.082128    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:28:05.116619 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:05.116565    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:28:15.081837 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:15.081702    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:28:15.115818 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:15.115768    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:28:25.081696 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:25.081642    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:28:25.116504 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:25.116456    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:28:35.081987 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:35.081940    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:28:35.116312 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:35.116258    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:28:45.081699 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:45.081643    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:28:45.116010 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:45.115959    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:28:45.686974 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:45.686938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:28:55.082297 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:55.082245    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:28:55.116411 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:55.116360    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:28:56.544383 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:28:56.544340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:29:05.082174 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:05.082119    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:29:05.115884 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:05.115837    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:29:15.082568 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:15.082518    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:29:15.116372 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:15.116329    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:29:25.082349 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:25.082283    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:29:25.115427 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:25.115378    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:29:35.081563 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:35.081509    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:29:35.115703 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:35.115650    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:29:45.081857 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:45.081748    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:29:45.116085 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:45.116039    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:29:55.081914 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:55.081861    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:29:55.116202 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:29:55.116151    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:30:04.642312 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:30:04.642267    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:30:04.642805 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:30:04.642417    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:30:05.082315 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:05.082265    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:30:05.116386 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:05.116344    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:30:05.240100 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:05.240073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:30:05.240268 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:05.240073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:30:15.081528 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:15.081471    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8001/health\": dial tcp 10.134.0.42:8001: connect: connection refused"
Jul 06 21:30:15.115973 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:15.115924    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 06 21:30:25.091533 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:25.091500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:30:25.109976 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:25.109949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:30:25.125621 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:25.125573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:30:25.134872 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:25.134842    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:30:40.822296 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:40.822250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"]
Jul 06 21:30:40.825778 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:40.825752    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"]
Jul 06 21:30:40.849958 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:40.849928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:30:40.917602 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:40.917571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:30:55.846870 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:55.846806    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main" containerID="cri-o://a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d" gracePeriod=45
Jul 06 21:30:55.848421 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:55.848390    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main" containerID="cri-o://03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71" gracePeriod=45
Jul 06 21:30:56.670124 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.670074    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="llm-d-routing-sidecar" containerID="cri-o://33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9" gracePeriod=45
Jul 06 21:30:56.812052 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.812022    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:30:56.923293 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.923269    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:30:56.930024 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.929999    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-model-cache\") pod \"da56c2c4-c212-413e-99a1-81708d53196b\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") "
Jul 06 21:30:56.930171 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da56c2c4-c212-413e-99a1-81708d53196b-tls-certs\") pod \"da56c2c4-c212-413e-99a1-81708d53196b\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") "
Jul 06 21:30:56.930171 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930085    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-home\") pod \"da56c2c4-c212-413e-99a1-81708d53196b\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") "
Jul 06 21:30:56.930171 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmn5g\" (UniqueName: \"kubernetes.io/projected/da56c2c4-c212-413e-99a1-81708d53196b-kube-api-access-kmn5g\") pod \"da56c2c4-c212-413e-99a1-81708d53196b\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") "
Jul 06 21:30:56.930338 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-tmp-dir\") pod \"da56c2c4-c212-413e-99a1-81708d53196b\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") "
Jul 06 21:30:56.930338 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-dshm\") pod \"da56c2c4-c212-413e-99a1-81708d53196b\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") "
Jul 06 21:30:56.930338 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-kserve-provision-location\") pod \"da56c2c4-c212-413e-99a1-81708d53196b\" (UID: \"da56c2c4-c212-413e-99a1-81708d53196b\") "
Jul 06 21:30:56.930338 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930274    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-model-cache" (OuterVolumeSpecName: "model-cache") pod "da56c2c4-c212-413e-99a1-81708d53196b" (UID: "da56c2c4-c212-413e-99a1-81708d53196b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:56.930593 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930574    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:56.930827 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.930800    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-home" (OuterVolumeSpecName: "home") pod "da56c2c4-c212-413e-99a1-81708d53196b" (UID: "da56c2c4-c212-413e-99a1-81708d53196b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:56.932568 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.932542    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da56c2c4-c212-413e-99a1-81708d53196b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da56c2c4-c212-413e-99a1-81708d53196b" (UID: "da56c2c4-c212-413e-99a1-81708d53196b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:30:56.932691 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.932662    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-dshm" (OuterVolumeSpecName: "dshm") pod "da56c2c4-c212-413e-99a1-81708d53196b" (UID: "da56c2c4-c212-413e-99a1-81708d53196b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:56.932850 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.932741    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da56c2c4-c212-413e-99a1-81708d53196b-kube-api-access-kmn5g" (OuterVolumeSpecName: "kube-api-access-kmn5g") pod "da56c2c4-c212-413e-99a1-81708d53196b" (UID: "da56c2c4-c212-413e-99a1-81708d53196b"). InnerVolumeSpecName "kube-api-access-kmn5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:30:56.943752 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.943707    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da56c2c4-c212-413e-99a1-81708d53196b" (UID: "da56c2c4-c212-413e-99a1-81708d53196b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:56.995569 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:56.995512    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da56c2c4-c212-413e-99a1-81708d53196b" (UID: "da56c2c4-c212-413e-99a1-81708d53196b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:57.031018 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.030978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-dshm\") pod \"b61ec122-85b1-41b5-874e-9f1f11e5d956\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") "
Jul 06 21:30:57.031161 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg4gv\" (UniqueName: \"kubernetes.io/projected/b61ec122-85b1-41b5-874e-9f1f11e5d956-kube-api-access-vg4gv\") pod \"b61ec122-85b1-41b5-874e-9f1f11e5d956\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") "
Jul 06 21:30:57.031161 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-home\") pod \"b61ec122-85b1-41b5-874e-9f1f11e5d956\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") "
Jul 06 21:30:57.031161 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-model-cache\") pod \"b61ec122-85b1-41b5-874e-9f1f11e5d956\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") "
Jul 06 21:30:57.031161 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec122-85b1-41b5-874e-9f1f11e5d956-tls-certs\") pod \"b61ec122-85b1-41b5-874e-9f1f11e5d956\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") "
Jul 06 21:30:57.031371 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-tmp-dir\") pod \"b61ec122-85b1-41b5-874e-9f1f11e5d956\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") "
Jul 06 21:30:57.031371 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031212    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-kserve-provision-location\") pod \"b61ec122-85b1-41b5-874e-9f1f11e5d956\" (UID: \"b61ec122-85b1-41b5-874e-9f1f11e5d956\") "
Jul 06 21:30:57.031477 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031398    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-model-cache" (OuterVolumeSpecName: "model-cache") pod "b61ec122-85b1-41b5-874e-9f1f11e5d956" (UID: "b61ec122-85b1-41b5-874e-9f1f11e5d956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:57.031536 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031500    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.031536 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031520    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.031536 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031532    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.031681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031546    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.031681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031566    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da56c2c4-c212-413e-99a1-81708d53196b-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.031681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031586    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da56c2c4-c212-413e-99a1-81708d53196b-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.031681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031599    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmn5g\" (UniqueName: \"kubernetes.io/projected/da56c2c4-c212-413e-99a1-81708d53196b-kube-api-access-kmn5g\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.031933 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.031908    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-home" (OuterVolumeSpecName: "home") pod "b61ec122-85b1-41b5-874e-9f1f11e5d956" (UID: "b61ec122-85b1-41b5-874e-9f1f11e5d956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:57.033285 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.033259    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ec122-85b1-41b5-874e-9f1f11e5d956-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b61ec122-85b1-41b5-874e-9f1f11e5d956" (UID: "b61ec122-85b1-41b5-874e-9f1f11e5d956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:30:57.033543 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.033515    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-dshm" (OuterVolumeSpecName: "dshm") pod "b61ec122-85b1-41b5-874e-9f1f11e5d956" (UID: "b61ec122-85b1-41b5-874e-9f1f11e5d956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:57.033845 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.033814    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61ec122-85b1-41b5-874e-9f1f11e5d956-kube-api-access-vg4gv" (OuterVolumeSpecName: "kube-api-access-vg4gv") pod "b61ec122-85b1-41b5-874e-9f1f11e5d956" (UID: "b61ec122-85b1-41b5-874e-9f1f11e5d956"). InnerVolumeSpecName "kube-api-access-vg4gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:30:57.043464 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.043437    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b61ec122-85b1-41b5-874e-9f1f11e5d956" (UID: "b61ec122-85b1-41b5-874e-9f1f11e5d956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:57.098069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.097966    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b61ec122-85b1-41b5-874e-9f1f11e5d956" (UID: "b61ec122-85b1-41b5-874e-9f1f11e5d956"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:30:57.132409 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.132364    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4gv\" (UniqueName: \"kubernetes.io/projected/b61ec122-85b1-41b5-874e-9f1f11e5d956-kube-api-access-vg4gv\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.132409 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.132402    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.132409 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.132415    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec122-85b1-41b5-874e-9f1f11e5d956-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.132650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.132429    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.132650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.132441    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.132650 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.132452    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec122-85b1-41b5-874e-9f1f11e5d956-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:30:57.484027 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.483940    2578 generic.go:356] "Generic (PLEG): container finished" podID="da56c2c4-c212-413e-99a1-81708d53196b" containerID="03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71" exitCode=0
Jul 06 21:30:57.484027 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.483997    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"
Jul 06 21:30:57.484234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.484028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" event={"ID":"da56c2c4-c212-413e-99a1-81708d53196b","Type":"ContainerDied","Data":"03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71"}
Jul 06 21:30:57.484234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.484090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc" event={"ID":"da56c2c4-c212-413e-99a1-81708d53196b","Type":"ContainerDied","Data":"18d60159d7e7bb3eb5ace0a1a347bfce18eaa5193b03352fa559c52725cfcf1a"}
Jul 06 21:30:57.484234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.484121    2578 scope.go:117] "RemoveContainer" containerID="03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71"
Jul 06 21:30:57.485993 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.485967    2578 generic.go:356] "Generic (PLEG): container finished" podID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerID="a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d" exitCode=0
Jul 06 21:30:57.485993 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.485990    2578 generic.go:356] "Generic (PLEG): container finished" podID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerID="33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9" exitCode=0
Jul 06 21:30:57.486149 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.486019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerDied","Data":"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"}
Jul 06 21:30:57.486149 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.486040    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"
Jul 06 21:30:57.486149 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.486061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerDied","Data":"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"}
Jul 06 21:30:57.486149 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.486082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l" event={"ID":"b61ec122-85b1-41b5-874e-9f1f11e5d956","Type":"ContainerDied","Data":"534d692480a220cdfbab0b86b1a8163053c768046c5294c8fa1fb887b4fcf5a5"}
Jul 06 21:30:57.493887 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.493865    2578 scope.go:117] "RemoveContainer" containerID="fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6"
Jul 06 21:30:57.502317 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.502295    2578 scope.go:117] "RemoveContainer" containerID="03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71"
Jul 06 21:30:57.502620 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:30:57.502597    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71\": container with ID starting with 03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71 not found: ID does not exist" containerID="03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71"
Jul 06 21:30:57.502697 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.502630    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71"} err="failed to get container status \"03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71\": rpc error: code = NotFound desc = could not find container \"03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71\": container with ID starting with 03519e9b11b4722edf7bcc9040c8a4017ddf5bcc4e6a7a8f1ae5a4dbdd34fe71 not found: ID does not exist"
Jul 06 21:30:57.502697 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.502648    2578 scope.go:117] "RemoveContainer" containerID="fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6"
Jul 06 21:30:57.502912 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:30:57.502895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6\": container with ID starting with fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6 not found: ID does not exist" containerID="fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6"
Jul 06 21:30:57.502951 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.502917    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6"} err="failed to get container status \"fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6\": rpc error: code = NotFound desc = could not find container \"fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6\": container with ID starting with fd13790f3ec6e507b76cc543f047b9bcbf1abab2dbb2c8ab45f411d5c143bad6 not found: ID does not exist"
Jul 06 21:30:57.502951 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.502931    2578 scope.go:117] "RemoveContainer" containerID="a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"
Jul 06 21:30:57.507315 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.507284    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"]
Jul 06 21:30:57.510840 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.510818    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc"]
Jul 06 21:30:57.511827 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.511754    2578 scope.go:117] "RemoveContainer" containerID="84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"
Jul 06 21:30:57.519779 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.519759    2578 scope.go:117] "RemoveContainer" containerID="33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"
Jul 06 21:30:57.522262 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.522236    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"]
Jul 06 21:30:57.528143 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.528123    2578 scope.go:117] "RemoveContainer" containerID="a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"
Jul 06 21:30:57.528200 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.528142    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l"]
Jul 06 21:30:57.528414 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:30:57.528395    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d\": container with ID starting with a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d not found: ID does not exist" containerID="a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"
Jul 06 21:30:57.528473 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.528420    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"} err="failed to get container status \"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d\": rpc error: code = NotFound desc = could not find container \"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d\": container with ID starting with a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d not found: ID does not exist"
Jul 06 21:30:57.528473 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.528439    2578 scope.go:117] "RemoveContainer" containerID="84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"
Jul 06 21:30:57.528692 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:30:57.528674    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e\": container with ID starting with 84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e not found: ID does not exist" containerID="84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"
Jul 06 21:30:57.528753 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.528698    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"} err="failed to get container status \"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e\": rpc error: code = NotFound desc = could not find container \"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e\": container with ID starting with 84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e not found: ID does not exist"
Jul 06 21:30:57.528753 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.528732    2578 scope.go:117] "RemoveContainer" containerID="33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"
Jul 06 21:30:57.528980 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:30:57.528963    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9\": container with ID starting with 33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9 not found: ID does not exist" containerID="33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"
Jul 06 21:30:57.529045 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.528989    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"} err="failed to get container status \"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9\": rpc error: code = NotFound desc = could not find container \"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9\": container with ID starting with 33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9 not found: ID does not exist"
Jul 06 21:30:57.529045 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.529011    2578 scope.go:117] "RemoveContainer" containerID="a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"
Jul 06 21:30:57.529244 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.529225    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d"} err="failed to get container status \"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d\": rpc error: code = NotFound desc = could not find container \"a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d\": container with ID starting with a0c9d4906f704893dfad163a111cc26eec9e7f6bb3f4ac4e53fc05f6e77f3c2d not found: ID does not exist"
Jul 06 21:30:57.529296 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.529245    2578 scope.go:117] "RemoveContainer" containerID="84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"
Jul 06 21:30:57.529432 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.529414    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e"} err="failed to get container status \"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e\": rpc error: code = NotFound desc = could not find container \"84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e\": container with ID starting with 84ae9492f6a5a08a2c41a67acd586aa63974a1b08fb6628d4b13cabbeb96876e not found: ID does not exist"
Jul 06 21:30:57.529474 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.529432    2578 scope.go:117] "RemoveContainer" containerID="33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"
Jul 06 21:30:57.529641 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:57.529623    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9"} err="failed to get container status \"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9\": rpc error: code = NotFound desc = could not find container \"33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9\": container with ID starting with 33b4a0f09e02360216f3acbd7fd8ffb43308dc3e3ffca8b7a29655d4397b6ce9 not found: ID does not exist"
Jul 06 21:30:58.160322 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:58.160288    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" path="/var/lib/kubelet/pods/b61ec122-85b1-41b5-874e-9f1f11e5d956/volumes"
Jul 06 21:30:58.160777 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:30:58.160762    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da56c2c4-c212-413e-99a1-81708d53196b" path="/var/lib/kubelet/pods/da56c2c4-c212-413e-99a1-81708d53196b/volumes"
Jul 06 21:31:00.454005 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:00.453965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:31:00.550633 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:00.550597    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:31:04.592395 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:04.592361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:31:39.582698 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.582658    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"]
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583000    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="llm-d-routing-sidecar"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583013    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="llm-d-routing-sidecar"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583029    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583040    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583045    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583053    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="storage-initializer"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583058    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="storage-initializer"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583067    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="storage-initializer"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583072    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="storage-initializer"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583121    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="llm-d-routing-sidecar"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583129    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da56c2c4-c212-413e-99a1-81708d53196b" containerName="main"
Jul 06 21:31:39.583234 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.583135    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61ec122-85b1-41b5-874e-9f1f11e5d956" containerName="main"
Jul 06 21:31:39.585674 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.585657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.588973 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.588951    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-5zttx\""
Jul 06 21:31:39.589137 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.589125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:31:39.589470 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.589453    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 06 21:31:39.605968 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.605931    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"]
Jul 06 21:31:39.609427 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.609402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.609613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.609593    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"]
Jul 06 21:31:39.618365 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.618339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.618495 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.618379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.618495 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.618411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-dshm\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.618606 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.618510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vb5\" (UniqueName: \"kubernetes.io/projected/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kube-api-access-88vb5\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.618606 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.618553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-home\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.618693 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.618610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-model-cache\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.618693 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.618638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.625158 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.625123    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"]
Jul 06 21:31:39.665531 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.665497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:31:39.719288 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.719490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-dshm\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.719490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vb5\" (UniqueName: \"kubernetes.io/projected/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kube-api-access-88vb5\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.719490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.719490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rd7\" (UniqueName: \"kubernetes.io/projected/b93822ef-d232-4e16-9712-50dec9c9ae6a-kube-api-access-42rd7\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.719490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-home\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.719490 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.719846 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-home\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.719846 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93822ef-d232-4e16-9712-50dec9c9ae6a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.719846 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-model-cache\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.719846 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.719846 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.719846 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.720071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.720071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719887    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-home\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.720071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.719981    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.720071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.720012    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-model-cache\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.720071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.720032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.721729 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.721698    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-dshm\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.722054 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.722038    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.728218 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.728192    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vb5\" (UniqueName: \"kubernetes.io/projected/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kube-api-access-88vb5\") pod \"router-with-refs-pd-test-kserve-5f4558569-7l4qq\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.820570 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.820530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.820787 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.820661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.820787 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.820762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.820886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.820785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rd7\" (UniqueName: \"kubernetes.io/projected/b93822ef-d232-4e16-9712-50dec9c9ae6a-kube-api-access-42rd7\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.820886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.820813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.820886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.820843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-home\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.820886 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.820868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93822ef-d232-4e16-9712-50dec9c9ae6a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.821082 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.821053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.821155 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.821133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.821215 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.821194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-home\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.821299 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.821276    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.822914 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.822893    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.823216 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.823197    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93822ef-d232-4e16-9712-50dec9c9ae6a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.829786 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.829762    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rd7\" (UniqueName: \"kubernetes.io/projected/b93822ef-d232-4e16-9712-50dec9c9ae6a-kube-api-access-42rd7\") pod \"router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:39.895844 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.895756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:39.920396 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:39.920362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:40.052320 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.052134    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"]
Jul 06 21:31:40.054521 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:31:40.054479    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcf0b3d_948b_453d_a47d_f2e3da2d430d.slice/crio-69cc3b6ceab1d2fa329d4ab3248f7531c3a4536982aefc9b8fd710f325ed0bcd WatchSource:0}: Error finding container 69cc3b6ceab1d2fa329d4ab3248f7531c3a4536982aefc9b8fd710f325ed0bcd: Status 404 returned error can't find the container with id 69cc3b6ceab1d2fa329d4ab3248f7531c3a4536982aefc9b8fd710f325ed0bcd
Jul 06 21:31:40.070489 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.070435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"]
Jul 06 21:31:40.072682 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:31:40.072657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93822ef_d232_4e16_9712_50dec9c9ae6a.slice/crio-009b9877c0db10d5fa045eefe6f59b344f4a4853ea9df05ef2506d91d2dd43fc WatchSource:0}: Error finding container 009b9877c0db10d5fa045eefe6f59b344f4a4853ea9df05ef2506d91d2dd43fc: Status 404 returned error can't find the container with id 009b9877c0db10d5fa045eefe6f59b344f4a4853ea9df05ef2506d91d2dd43fc
Jul 06 21:31:40.315249 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.315211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:31:40.420071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.420039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:31:40.674448 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.674361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" event={"ID":"b93822ef-d232-4e16-9712-50dec9c9ae6a","Type":"ContainerStarted","Data":"ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af"}
Jul 06 21:31:40.674448 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.674427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" event={"ID":"b93822ef-d232-4e16-9712-50dec9c9ae6a","Type":"ContainerStarted","Data":"009b9877c0db10d5fa045eefe6f59b344f4a4853ea9df05ef2506d91d2dd43fc"}
Jul 06 21:31:40.675964 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.675941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerStarted","Data":"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"}
Jul 06 21:31:40.676068 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.675989    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:40.676068 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:40.676008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerStarted","Data":"69cc3b6ceab1d2fa329d4ab3248f7531c3a4536982aefc9b8fd710f325ed0bcd"}
Jul 06 21:31:41.691764 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:41.691708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerStarted","Data":"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"}
Jul 06 21:31:44.710005 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:44.709972    2578 generic.go:356] "Generic (PLEG): container finished" podID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerID="ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af" exitCode=0
Jul 06 21:31:44.710392 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:44.710054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" event={"ID":"b93822ef-d232-4e16-9712-50dec9c9ae6a","Type":"ContainerDied","Data":"ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af"}
Jul 06 21:31:45.716845 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:45.716804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" event={"ID":"b93822ef-d232-4e16-9712-50dec9c9ae6a","Type":"ContainerStarted","Data":"d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba"}
Jul 06 21:31:45.718657 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:45.718630    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerID="de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42" exitCode=0
Jul 06 21:31:45.718813 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:45.718672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerDied","Data":"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"}
Jul 06 21:31:45.743555 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:45.743507    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podStartSLOduration=2.653028713 podStartE2EDuration="6.743489211s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.090460498s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:31:45.740962372 +0000 UTC m=+2548.200724248" watchObservedRunningTime="2026-07-06 21:31:45.743489211 +0000 UTC m=+2548.203251089"
Jul 06 21:31:46.726496 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:46.726458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerStarted","Data":"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"}
Jul 06 21:31:46.753896 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:46.753824    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podStartSLOduration=3.347534057 podStartE2EDuration="7.753800868s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.406266811s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:31:46.752526876 +0000 UTC m=+2549.212288766" watchObservedRunningTime="2026-07-06 21:31:46.753800868 +0000 UTC m=+2549.213562747"
Jul 06 21:31:49.896613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:49.896573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:49.897067 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:49.896648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:49.898214 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:49.898186    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:31:49.920876 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:49.920834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:49.921068 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:49.920920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:31:49.922528 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:49.922492    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:31:59.896405 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:59.896349    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:31:59.915173 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:59.915142    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:31:59.921009 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:31:59.920968    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:32:08.241695 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:32:08.241642    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:32:08.241695 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:32:08.241642    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:32:08.833195 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:08.833161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:32:08.833383 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:08.833163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:32:09.896323 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:09.896265    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:32:09.921092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:09.921049    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:32:19.896681 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:19.896629    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:32:19.921158 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:19.921118    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:32:29.896951 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:29.896895    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:32:29.921157 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:29.921111    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:32:39.896773 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:39.896647    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:32:39.920905 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:39.920860    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:32:49.897202 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:49.897144    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:32:49.921439 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:49.921395    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:32:54.507339 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:54.507303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:32:55.946333 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:55.946293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:32:59.896900 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:59.896834    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:32:59.920956 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:32:59.920906    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:33:09.896310 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:09.896258    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:33:09.920695 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:09.920650    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:33:19.896956 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:19.896898    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:33:19.920819 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:19.920769    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:33:29.896321 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:29.896263    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:33:29.920920 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:29.920873    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:33:39.896978 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:39.896925    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:33:39.921205 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:39.921166    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:33:49.896764 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:49.896696    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:33:49.920906 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:49.920869    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:33:59.896613 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:59.896560    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 06 21:33:59.920751 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:33:59.920701    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:34:09.913019 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:09.912930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:34:09.920819 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:09.920774    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:34:09.925691 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:09.925666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:34:11.834351 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:34:11.834303    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:34:11.834768 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:34:11.834342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:34:12.422683 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:12.422654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:34:12.422683 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:12.422691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:34:19.921241 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:19.921197    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 06 21:34:22.477994 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:22.477962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:34:29.933512 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:29.933478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:34:29.941240 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:34:29.941213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:35:08.053041 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:08.053005    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"]
Jul 06 21:35:08.057449 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:08.057415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:35:08.058357 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:08.058334    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"]
Jul 06 21:35:08.131286 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:08.131246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:35:12.135141 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:12.135104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:35:23.077400 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:23.077343    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main" containerID="cri-o://d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba" gracePeriod=45
Jul 06 21:35:23.083545 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:23.083486    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main" containerID="cri-o://08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61" gracePeriod=45
Jul 06 21:35:23.900218 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:23.900164    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="llm-d-routing-sidecar" containerID="cri-o://ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3" gracePeriod=45
Jul 06 21:35:24.028836 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.028804    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:35:24.146924 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.146896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93822ef-d232-4e16-9712-50dec9c9ae6a-tls-certs\") pod \"b93822ef-d232-4e16-9712-50dec9c9ae6a\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") "
Jul 06 21:35:24.147318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.146939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42rd7\" (UniqueName: \"kubernetes.io/projected/b93822ef-d232-4e16-9712-50dec9c9ae6a-kube-api-access-42rd7\") pod \"b93822ef-d232-4e16-9712-50dec9c9ae6a\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") "
Jul 06 21:35:24.147318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.146971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-model-cache\") pod \"b93822ef-d232-4e16-9712-50dec9c9ae6a\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") "
Jul 06 21:35:24.147318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.146994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-dshm\") pod \"b93822ef-d232-4e16-9712-50dec9c9ae6a\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") "
Jul 06 21:35:24.147318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.147062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-kserve-provision-location\") pod \"b93822ef-d232-4e16-9712-50dec9c9ae6a\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") "
Jul 06 21:35:24.147318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.147076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-tmp-dir\") pod \"b93822ef-d232-4e16-9712-50dec9c9ae6a\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") "
Jul 06 21:35:24.147318 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.147107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-home\") pod \"b93822ef-d232-4e16-9712-50dec9c9ae6a\" (UID: \"b93822ef-d232-4e16-9712-50dec9c9ae6a\") "
Jul 06 21:35:24.147585 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.147326    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b93822ef-d232-4e16-9712-50dec9c9ae6a" (UID: "b93822ef-d232-4e16-9712-50dec9c9ae6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.147956 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.147917    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-home" (OuterVolumeSpecName: "home") pod "b93822ef-d232-4e16-9712-50dec9c9ae6a" (UID: "b93822ef-d232-4e16-9712-50dec9c9ae6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.148091 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.147973    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:35:24.149312 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.149284    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-dshm" (OuterVolumeSpecName: "dshm") pod "b93822ef-d232-4e16-9712-50dec9c9ae6a" (UID: "b93822ef-d232-4e16-9712-50dec9c9ae6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.149524 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.149499    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93822ef-d232-4e16-9712-50dec9c9ae6a-kube-api-access-42rd7" (OuterVolumeSpecName: "kube-api-access-42rd7") pod "b93822ef-d232-4e16-9712-50dec9c9ae6a" (UID: "b93822ef-d232-4e16-9712-50dec9c9ae6a"). InnerVolumeSpecName "kube-api-access-42rd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:35:24.149618 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.149550    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93822ef-d232-4e16-9712-50dec9c9ae6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b93822ef-d232-4e16-9712-50dec9c9ae6a" (UID: "b93822ef-d232-4e16-9712-50dec9c9ae6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:35:24.159439 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.159403    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b93822ef-d232-4e16-9712-50dec9c9ae6a" (UID: "b93822ef-d232-4e16-9712-50dec9c9ae6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.223500 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.223450    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b93822ef-d232-4e16-9712-50dec9c9ae6a" (UID: "b93822ef-d232-4e16-9712-50dec9c9ae6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.247811 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.247778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kserve-provision-location\") pod \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") "
Jul 06 21:35:24.247925 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.247820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tmp-dir\") pod \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") "
Jul 06 21:35:24.247925 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.247867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88vb5\" (UniqueName: \"kubernetes.io/projected/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kube-api-access-88vb5\") pod \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") "
Jul 06 21:35:24.248003 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.247942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-home\") pod \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") "
Jul 06 21:35:24.248056 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248004    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-dshm\") pod \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") "
Jul 06 21:35:24.248056 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tls-certs\") pod \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") "
Jul 06 21:35:24.248161 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-model-cache\") pod \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\" (UID: \"8fcf0b3d-948b-453d-a47d-f2e3da2d430d\") "
Jul 06 21:35:24.248428 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248392    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.248428 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248418    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.248428 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248432    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.248746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248445    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93822ef-d232-4e16-9712-50dec9c9ae6a-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.248746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248461    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42rd7\" (UniqueName: \"kubernetes.io/projected/b93822ef-d232-4e16-9712-50dec9c9ae6a-kube-api-access-42rd7\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.248746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248473    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.248746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248486    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93822ef-d232-4e16-9712-50dec9c9ae6a-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.248746 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248635    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fcf0b3d-948b-453d-a47d-f2e3da2d430d" (UID: "8fcf0b3d-948b-453d-a47d-f2e3da2d430d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.249032 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.248752    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-home" (OuterVolumeSpecName: "home") pod "8fcf0b3d-948b-453d-a47d-f2e3da2d430d" (UID: "8fcf0b3d-948b-453d-a47d-f2e3da2d430d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.250227 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.250197    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-dshm" (OuterVolumeSpecName: "dshm") pod "8fcf0b3d-948b-453d-a47d-f2e3da2d430d" (UID: "8fcf0b3d-948b-453d-a47d-f2e3da2d430d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.250466 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.250447    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fcf0b3d-948b-453d-a47d-f2e3da2d430d" (UID: "8fcf0b3d-948b-453d-a47d-f2e3da2d430d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:35:24.250582 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.250560    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kube-api-access-88vb5" (OuterVolumeSpecName: "kube-api-access-88vb5") pod "8fcf0b3d-948b-453d-a47d-f2e3da2d430d" (UID: "8fcf0b3d-948b-453d-a47d-f2e3da2d430d"). InnerVolumeSpecName "kube-api-access-88vb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:35:24.261643 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.261618    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fcf0b3d-948b-453d-a47d-f2e3da2d430d" (UID: "8fcf0b3d-948b-453d-a47d-f2e3da2d430d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.313845 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.313802    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fcf0b3d-948b-453d-a47d-f2e3da2d430d" (UID: "8fcf0b3d-948b-453d-a47d-f2e3da2d430d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:24.350061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.349980    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.350061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.350012    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.350061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.350021    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.350061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.350031    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.350061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.350043    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.350061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.350055    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.350061 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.350063    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88vb5\" (UniqueName: \"kubernetes.io/projected/8fcf0b3d-948b-453d-a47d-f2e3da2d430d-kube-api-access-88vb5\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:35:24.751641 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.751541    2578 generic.go:356] "Generic (PLEG): container finished" podID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerID="d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba" exitCode=0
Jul 06 21:35:24.751641 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.751602    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"
Jul 06 21:35:24.751641 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.751630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" event={"ID":"b93822ef-d232-4e16-9712-50dec9c9ae6a","Type":"ContainerDied","Data":"d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba"}
Jul 06 21:35:24.751966 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.751695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f" event={"ID":"b93822ef-d232-4e16-9712-50dec9c9ae6a","Type":"ContainerDied","Data":"009b9877c0db10d5fa045eefe6f59b344f4a4853ea9df05ef2506d91d2dd43fc"}
Jul 06 21:35:24.751966 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.751762    2578 scope.go:117] "RemoveContainer" containerID="d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba"
Jul 06 21:35:24.753600 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.753567    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerID="08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61" exitCode=0
Jul 06 21:35:24.753600 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.753582    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerID="ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3" exitCode=0
Jul 06 21:35:24.753896 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.753595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerDied","Data":"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"}
Jul 06 21:35:24.753896 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.753625    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"
Jul 06 21:35:24.753896 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.753652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerDied","Data":"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"}
Jul 06 21:35:24.753896 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.753680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq" event={"ID":"8fcf0b3d-948b-453d-a47d-f2e3da2d430d","Type":"ContainerDied","Data":"69cc3b6ceab1d2fa329d4ab3248f7531c3a4536982aefc9b8fd710f325ed0bcd"}
Jul 06 21:35:24.760695 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.760656    2578 scope.go:117] "RemoveContainer" containerID="ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af"
Jul 06 21:35:24.768937 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.768920    2578 scope.go:117] "RemoveContainer" containerID="d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba"
Jul 06 21:35:24.769198 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:35:24.769178    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba\": container with ID starting with d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba not found: ID does not exist" containerID="d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba"
Jul 06 21:35:24.769277 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.769211    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba"} err="failed to get container status \"d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba\": rpc error: code = NotFound desc = could not find container \"d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba\": container with ID starting with d2b998d11fbe2feb58ecd2b5850f157a2ea66c0736997096199f0207fac34eba not found: ID does not exist"
Jul 06 21:35:24.769277 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.769238    2578 scope.go:117] "RemoveContainer" containerID="ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af"
Jul 06 21:35:24.769502 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:35:24.769483    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af\": container with ID starting with ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af not found: ID does not exist" containerID="ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af"
Jul 06 21:35:24.769543 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.769508    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af"} err="failed to get container status \"ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af\": rpc error: code = NotFound desc = could not find container \"ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af\": container with ID starting with ac5560f8e8db9ec71ebbbd17a1029bc151353056416ed5bf955063b013d419af not found: ID does not exist"
Jul 06 21:35:24.769543 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.769523    2578 scope.go:117] "RemoveContainer" containerID="08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"
Jul 06 21:35:24.779026 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.778483    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"]
Jul 06 21:35:24.779026 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.778680    2578 scope.go:117] "RemoveContainer" containerID="de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"
Jul 06 21:35:24.784060 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.784005    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq"]
Jul 06 21:35:24.789541 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.789518    2578 scope.go:117] "RemoveContainer" containerID="ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"
Jul 06 21:35:24.795060 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.795037    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"]
Jul 06 21:35:24.800595 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.800572    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f"]
Jul 06 21:35:24.806071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.806052    2578 scope.go:117] "RemoveContainer" containerID="08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"
Jul 06 21:35:24.806360 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:35:24.806343    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61\": container with ID starting with 08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61 not found: ID does not exist" containerID="08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"
Jul 06 21:35:24.806408 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.806367    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"} err="failed to get container status \"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61\": rpc error: code = NotFound desc = could not find container \"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61\": container with ID starting with 08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61 not found: ID does not exist"
Jul 06 21:35:24.806408 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.806387    2578 scope.go:117] "RemoveContainer" containerID="de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"
Jul 06 21:35:24.806650 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:35:24.806632    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42\": container with ID starting with de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42 not found: ID does not exist" containerID="de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"
Jul 06 21:35:24.806693 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.806657    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"} err="failed to get container status \"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42\": rpc error: code = NotFound desc = could not find container \"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42\": container with ID starting with de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42 not found: ID does not exist"
Jul 06 21:35:24.806693 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.806675    2578 scope.go:117] "RemoveContainer" containerID="ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"
Jul 06 21:35:24.806971 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:35:24.806953    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3\": container with ID starting with ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3 not found: ID does not exist" containerID="ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"
Jul 06 21:35:24.807020 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.806980    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"} err="failed to get container status \"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3\": rpc error: code = NotFound desc = could not find container \"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3\": container with ID starting with ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3 not found: ID does not exist"
Jul 06 21:35:24.807020 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.807001    2578 scope.go:117] "RemoveContainer" containerID="08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"
Jul 06 21:35:24.807232 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.807214    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61"} err="failed to get container status \"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61\": rpc error: code = NotFound desc = could not find container \"08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61\": container with ID starting with 08ba73a9b14c9c41694b4a4e548595ce9cb8745660f8f43ffa34a19c364bbf61 not found: ID does not exist"
Jul 06 21:35:24.807275 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.807232    2578 scope.go:117] "RemoveContainer" containerID="de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"
Jul 06 21:35:24.807459 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.807440    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42"} err="failed to get container status \"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42\": rpc error: code = NotFound desc = could not find container \"de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42\": container with ID starting with de85cca344c3103a98c1c6344c04941c9edd6ff9f9badcb2ac534d8da72adf42 not found: ID does not exist"
Jul 06 21:35:24.807500 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.807460    2578 scope.go:117] "RemoveContainer" containerID="ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"
Jul 06 21:35:24.807649 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:24.807631    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3"} err="failed to get container status \"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3\": rpc error: code = NotFound desc = could not find container \"ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3\": container with ID starting with ee0ae5e96b8704f91d1e0348d6dded646776351349b52595cb4f17c574be65f3 not found: ID does not exist"
Jul 06 21:35:26.165962 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:26.165919    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" path="/var/lib/kubelet/pods/8fcf0b3d-948b-453d-a47d-f2e3da2d430d/volumes"
Jul 06 21:35:26.166614 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:26.166592    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" path="/var/lib/kubelet/pods/b93822ef-d232-4e16-9712-50dec9c9ae6a/volumes"
Jul 06 21:35:49.517627 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517552    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"]
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517874    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="llm-d-routing-sidecar"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517890    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="llm-d-routing-sidecar"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="storage-initializer"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="storage-initializer"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517916    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517925    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517935    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517943    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517964    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="storage-initializer"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.517971    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="storage-initializer"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.518019    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="main"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.518027    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93822ef-d232-4e16-9712-50dec9c9ae6a" containerName="main"
Jul 06 21:35:49.518092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.518034    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fcf0b3d-948b-453d-a47d-f2e3da2d430d" containerName="llm-d-routing-sidecar"
Jul 06 21:35:49.520937 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.520920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.523477 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.523457    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:35:49.523594 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.523517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 06 21:35:49.532731 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.532693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"]
Jul 06 21:35:49.545049 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.545020    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:35:49.575206 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.575165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.575206 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.575208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.575418 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.575232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.575418 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.575370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.575418 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.575405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.575556 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.575426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.575556 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.575451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhmm\" (UniqueName: \"kubernetes.io/projected/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kube-api-access-hqhmm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.676911 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.676867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.676911 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.676914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677175 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.676939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677175 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhmm\" (UniqueName: \"kubernetes.io/projected/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kube-api-access-hqhmm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677370 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677327    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677370 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677486 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677486 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677486 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677486 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.677686 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.677557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.679729 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.679690    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.679967 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.679950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.685092 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.685070    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhmm\" (UniqueName: \"kubernetes.io/projected/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kube-api-access-hqhmm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.831386 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.831346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:49.984620 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:49.984594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"]
Jul 06 21:35:49.987418 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:35:49.987392    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1ccb6e_8f0c_443d_a556_e2e1f2585179.slice/crio-14ea18066ecd646e63ddfbf769837009147c5949919d8279ab00b4d6cbd52505 WatchSource:0}: Error finding container 14ea18066ecd646e63ddfbf769837009147c5949919d8279ab00b4d6cbd52505: Status 404 returned error can't find the container with id 14ea18066ecd646e63ddfbf769837009147c5949919d8279ab00b4d6cbd52505
Jul 06 21:35:50.863583 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:50.863545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" event={"ID":"da1ccb6e-8f0c-443d-a556-e2e1f2585179","Type":"ContainerStarted","Data":"d00181de9591a56ffa69b1b5e478d5fa8c8040d7de7d7cb29e9c5f5acb4b808b"}
Jul 06 21:35:50.864105 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:50.863612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" event={"ID":"da1ccb6e-8f0c-443d-a556-e2e1f2585179","Type":"ContainerStarted","Data":"14ea18066ecd646e63ddfbf769837009147c5949919d8279ab00b4d6cbd52505"}
Jul 06 21:35:54.881669 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:54.881630    2578 generic.go:356] "Generic (PLEG): container finished" podID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerID="d00181de9591a56ffa69b1b5e478d5fa8c8040d7de7d7cb29e9c5f5acb4b808b" exitCode=0
Jul 06 21:35:54.882095 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:54.881703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" event={"ID":"da1ccb6e-8f0c-443d-a556-e2e1f2585179","Type":"ContainerDied","Data":"d00181de9591a56ffa69b1b5e478d5fa8c8040d7de7d7cb29e9c5f5acb4b808b"}
Jul 06 21:35:55.888755 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:55.888696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" event={"ID":"da1ccb6e-8f0c-443d-a556-e2e1f2585179","Type":"ContainerStarted","Data":"76e545af4d4c3b21529768d8ce7d2226ed3b4a9452e8a24123eff23aa16ceb97"}
Jul 06 21:35:55.914792 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:55.914711    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podStartSLOduration=2.62488785 podStartE2EDuration="6.914688875s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.289801025s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:35:55.912106458 +0000 UTC m=+2798.371868334" watchObservedRunningTime="2026-07-06 21:35:55.914688875 +0000 UTC m=+2798.374450751"
Jul 06 21:35:56.831860 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:56.831815    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:59.832510 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:59.832462    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:35:59.834293 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:35:59.834263    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:36:09.831972 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:09.831918    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:36:15.200674 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:15.200637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:36:15.300521 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:15.300489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:36:15.424429 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:36:15.424378    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:36:15.424429 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:36:15.424383    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:36:15.980516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:15.980486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:36:15.980705 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:15.980486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:36:19.832527 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:19.832477    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:36:29.832678 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:29.832625    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:36:39.832193 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:39.832152    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:36:49.832502 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:49.832454    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:36:59.832582 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:36:59.832529    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:37:09.832680 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:37:09.832594    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:37:19.832763 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:37:19.832703    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:37:29.832693 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:37:29.832651    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:37:39.832359 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:37:39.832305    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 06 21:37:49.126162 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:37:49.126123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:37:49.842516 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:37:49.842481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:37:49.850366 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:37:49.850338    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:38:18.935244 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:18.935207    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"]
Jul 06 21:38:18.962773 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:18.962741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:38:18.981891 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:38:18.981855    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:38:18.981989 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:38:18.981854    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:38:19.520647 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:19.520606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:38:19.520647 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:19.520635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:38:33.957959 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:33.957914    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main" containerID="cri-o://76e545af4d4c3b21529768d8ce7d2226ed3b4a9452e8a24123eff23aa16ceb97" gracePeriod=45
Jul 06 21:38:34.586032 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.585984    2578 generic.go:356] "Generic (PLEG): container finished" podID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerID="76e545af4d4c3b21529768d8ce7d2226ed3b4a9452e8a24123eff23aa16ceb97" exitCode=0
Jul 06 21:38:34.586215 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.586048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" event={"ID":"da1ccb6e-8f0c-443d-a556-e2e1f2585179","Type":"ContainerDied","Data":"76e545af4d4c3b21529768d8ce7d2226ed3b4a9452e8a24123eff23aa16ceb97"}
Jul 06 21:38:34.718990 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.718964    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:38:34.860069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kserve-provision-location\") pod \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") "
Jul 06 21:38:34.860069 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-dshm\") pod \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") "
Jul 06 21:38:34.860320 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-home\") pod \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") "
Jul 06 21:38:34.860320 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tmp-dir\") pod \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") "
Jul 06 21:38:34.860320 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqhmm\" (UniqueName: \"kubernetes.io/projected/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kube-api-access-hqhmm\") pod \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") "
Jul 06 21:38:34.860320 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tls-certs\") pod \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") "
Jul 06 21:38:34.860588 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-model-cache\") pod \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\" (UID: \"da1ccb6e-8f0c-443d-a556-e2e1f2585179\") "
Jul 06 21:38:34.860640 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860597    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-model-cache" (OuterVolumeSpecName: "model-cache") pod "da1ccb6e-8f0c-443d-a556-e2e1f2585179" (UID: "da1ccb6e-8f0c-443d-a556-e2e1f2585179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:38:34.860802 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860782    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-model-cache\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:38:34.860945 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.860917    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-home" (OuterVolumeSpecName: "home") pod "da1ccb6e-8f0c-443d-a556-e2e1f2585179" (UID: "da1ccb6e-8f0c-443d-a556-e2e1f2585179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:38:34.862547 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.862508    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kube-api-access-hqhmm" (OuterVolumeSpecName: "kube-api-access-hqhmm") pod "da1ccb6e-8f0c-443d-a556-e2e1f2585179" (UID: "da1ccb6e-8f0c-443d-a556-e2e1f2585179"). InnerVolumeSpecName "kube-api-access-hqhmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:38:34.862655 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.862598    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da1ccb6e-8f0c-443d-a556-e2e1f2585179" (UID: "da1ccb6e-8f0c-443d-a556-e2e1f2585179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:38:34.862699 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.862673    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-dshm" (OuterVolumeSpecName: "dshm") pod "da1ccb6e-8f0c-443d-a556-e2e1f2585179" (UID: "da1ccb6e-8f0c-443d-a556-e2e1f2585179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:38:34.873800 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.873760    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da1ccb6e-8f0c-443d-a556-e2e1f2585179" (UID: "da1ccb6e-8f0c-443d-a556-e2e1f2585179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:38:34.928647 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.928584    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da1ccb6e-8f0c-443d-a556-e2e1f2585179" (UID: "da1ccb6e-8f0c-443d-a556-e2e1f2585179"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:38:34.961426 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.961381    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kserve-provision-location\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:38:34.961426 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.961417    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-dshm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:38:34.961426 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.961430    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-home\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:38:34.961923 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.961441    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tmp-dir\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:38:34.961923 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.961453    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhmm\" (UniqueName: \"kubernetes.io/projected/da1ccb6e-8f0c-443d-a556-e2e1f2585179-kube-api-access-hqhmm\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:38:34.961923 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:34.961464    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ccb6e-8f0c-443d-a556-e2e1f2585179-tls-certs\") on node \"ip-10-0-140-38.ec2.internal\" DevicePath \"\""
Jul 06 21:38:35.594358 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:35.594323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7" event={"ID":"da1ccb6e-8f0c-443d-a556-e2e1f2585179","Type":"ContainerDied","Data":"14ea18066ecd646e63ddfbf769837009147c5949919d8279ab00b4d6cbd52505"}
Jul 06 21:38:35.594563 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:35.594387    2578 scope.go:117] "RemoveContainer" containerID="76e545af4d4c3b21529768d8ce7d2226ed3b4a9452e8a24123eff23aa16ceb97"
Jul 06 21:38:35.594563 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:35.594334    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"
Jul 06 21:38:35.603432 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:35.603412    2578 scope.go:117] "RemoveContainer" containerID="d00181de9591a56ffa69b1b5e478d5fa8c8040d7de7d7cb29e9c5f5acb4b808b"
Jul 06 21:38:35.622151 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:35.622124    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"]
Jul 06 21:38:35.628324 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:35.628295    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7"]
Jul 06 21:38:36.163928 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:36.162005    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" path="/var/lib/kubelet/pods/da1ccb6e-8f0c-443d-a556-e2e1f2585179/volumes"
Jul 06 21:38:47.898908 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:47.898872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:38:47.956457 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:38:47.956423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:39:06.465094 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.465052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:39:06.517429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.517395    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:39:06.517773 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.517755    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="storage-initializer"
Jul 06 21:39:06.517773 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.517774    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="storage-initializer"
Jul 06 21:39:06.517919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.517787    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main"
Jul 06 21:39:06.517919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.517794    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main"
Jul 06 21:39:06.517919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.517862    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1ccb6e-8f0c-443d-a556-e2e1f2585179" containerName="main"
Jul 06 21:39:06.520859 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.520840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.525762 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.525739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:39:06.525900 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.525741    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 06 21:39:06.525900 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.525818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gkgq6\""
Jul 06 21:39:06.548203 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.548169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:39:06.565999 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.565965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:39:06.624669 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.624631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s299c\" (UniqueName: \"kubernetes.io/projected/868a686f-eacb-42d1-8b43-c4373014e1b1-kube-api-access-s299c\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.624669 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.624682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.624919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.624702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.624919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.624796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/868a686f-eacb-42d1-8b43-c4373014e1b1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.624919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.624855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.624919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.624890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.624919 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.624914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s299c\" (UniqueName: \"kubernetes.io/projected/868a686f-eacb-42d1-8b43-c4373014e1b1-kube-api-access-s299c\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725429 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725707 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/868a686f-eacb-42d1-8b43-c4373014e1b1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725707 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725707 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725930 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725906    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.725997 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.726050 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.725997    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.726050 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.726039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.726158 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.726067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.727840 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.727822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/868a686f-eacb-42d1-8b43-c4373014e1b1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.728112 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.728095    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/868a686f-eacb-42d1-8b43-c4373014e1b1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.736040 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.736019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s299c\" (UniqueName: \"kubernetes.io/projected/868a686f-eacb-42d1-8b43-c4373014e1b1-kube-api-access-s299c\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"868a686f-eacb-42d1-8b43-c4373014e1b1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.806009 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.805967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:39:06.832558 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.832520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:06.993176 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:06.993094    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:39:06.996861 ip-10-0-140-38 kubenswrapper[2578]: W0706 21:39:06.996829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868a686f_eacb_42d1_8b43_c4373014e1b1.slice/crio-45c759b6e0098635cd43074469fe2012b09314955a6e726a6c5423035d17ecab WatchSource:0}: Error finding container 45c759b6e0098635cd43074469fe2012b09314955a6e726a6c5423035d17ecab: Status 404 returned error can't find the container with id 45c759b6e0098635cd43074469fe2012b09314955a6e726a6c5423035d17ecab
Jul 06 21:39:07.732828 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:07.732789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"868a686f-eacb-42d1-8b43-c4373014e1b1","Type":"ContainerStarted","Data":"dda45ae613b7bfba60594f908a3ec971cff23fdeaefd33394dc8a4c5a68039e4"}
Jul 06 21:39:07.733236 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:07.732849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"868a686f-eacb-42d1-8b43-c4373014e1b1","Type":"ContainerStarted","Data":"45c759b6e0098635cd43074469fe2012b09314955a6e726a6c5423035d17ecab"}
Jul 06 21:39:11.751829 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:11.751789    2578 generic.go:356] "Generic (PLEG): container finished" podID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerID="dda45ae613b7bfba60594f908a3ec971cff23fdeaefd33394dc8a4c5a68039e4" exitCode=0
Jul 06 21:39:11.752372 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:11.751868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"868a686f-eacb-42d1-8b43-c4373014e1b1","Type":"ContainerDied","Data":"dda45ae613b7bfba60594f908a3ec971cff23fdeaefd33394dc8a4c5a68039e4"}
Jul 06 21:39:12.758593 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:12.758558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"868a686f-eacb-42d1-8b43-c4373014e1b1","Type":"ContainerStarted","Data":"7e553a3afb5654f98b06d89f87b15b32b603120683ee6221eabe9c60a1e1a9c5"}
Jul 06 21:39:12.783525 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:12.783467    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.660378547 podStartE2EDuration="6.783449309s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.123070762s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:39:12.781476642 +0000 UTC m=+2995.241238519" watchObservedRunningTime="2026-07-06 21:39:12.783449309 +0000 UTC m=+2995.243211206"
Jul 06 21:39:12.833622 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:12.833582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:16.833506 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:16.833465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:39:16.835187 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:16.835144    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:39:26.833316 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:26.833265    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:39:29.880777 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:29.880712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:39:36.833432 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:36.833372    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:39:46.833959 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:46.833908    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:39:56.833970 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:39:56.833926    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:40:06.833501 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:06.833451    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:40:16.833222 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:16.833104    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:40:22.521489 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:40:22.521423    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:40:22.521489 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:40:22.521440    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:40:23.091530 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:23.091500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:40:23.091779 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:23.091499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:40:26.833421 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:26.833374    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:40:36.833808 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:36.833753    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:40:46.834017 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:46.833960    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="868a686f-eacb-42d1-8b43-c4373014e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 21:40:56.842939 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:56.842906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:40:56.851114 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:40:56.851076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:42:14.821392 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:42:14.821344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:42:26.092369 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:42:26.092323    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:42:26.092369 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:42:26.092323    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:42:26.619045 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:42:26.619013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:42:26.619226 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:42:26.619013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:44:29.620198 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:44:29.620151    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:44:29.620198 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:44:29.620152    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:44:30.145273 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:44:30.145235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:44:30.145273 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:44:30.145263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:44:35.937023 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:44:35.936987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:45:47.465483 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:45:47.465446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:45:47.522326 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:45:47.522293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:46:33.146481 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:46:33.146437    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:46:33.146481 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:46:33.146449    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:46:33.674352 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:46:33.674322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:46:33.674556 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:46:33.674322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:47:24.091049 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:47:24.091016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:47:24.149025 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:47:24.148988    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:48:36.675313 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:48:36.675264    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:48:36.675313 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:48:36.675273    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:48:37.207441 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:48:37.207406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:48:37.207633 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:48:37.207576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:50:29.693538 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:50:29.693458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:50:40.208959 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:50:40.208860    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:50:40.208959 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:50:40.208870    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:50:40.734264 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:50:40.734234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:50:40.734445 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:50:40.734239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:52:12.143038 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:52:12.142958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:52:43.735299 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:52:43.735249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:52:43.735299 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:52:43.735249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:52:44.257750 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:52:44.257700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:52:44.257927 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:52:44.257705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:53:42.336852 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:53:42.336766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 21:53:42.348807 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:53:42.348768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 21:54:16.463171 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:54:16.463130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:16.759053 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:54:16.758965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:47.259184 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:54:47.259140    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:54:47.259642 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:54:47.259133    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:54:47.774325 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:54:47.774294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:54:47.774518 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:54:47.774295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:54:49.093522 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:54:49.093485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:58.045236 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:54:58.045198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:58.115202 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:54:58.115166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:55:35.335913 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:55:35.335877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:55:36.595842 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:55:36.595802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:56:25.031111 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:56:25.031071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:56:34.977000 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:56:34.976954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:56:35.098895 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:56:35.098859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:56:50.775408 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:56:50.775312    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:56:50.775408 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:56:50.775313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:56:51.317491 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:56:51.317451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:56:51.317682 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:56:51.317451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:57:16.239951 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:57:16.239910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:57:16.556071 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:57:16.556025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:57:46.943120 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:57:46.943081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:11.512786 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:11.512566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:11.581891 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:11.581859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:34.964817 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:34.964771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:48.202580 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:48.202544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:52.128242 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:52.128202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:52.498892 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:52.498801    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:54.318872 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:58:54.318826    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 21:58:54.318872 ip-10-0-140-38 kubenswrapper[2578]: E0706 21:58:54.318826    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 21:58:54.847460 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:54.847426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:58:54.847676 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:54.847430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:58:59.612419 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:58:59.612382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:26.910355 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:59:26.910316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:33.829246 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:59:33.829206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:33.929419 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:59:33.929383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:41.679628 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:59:41.679537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:46.990635 ip-10-0-140-38 kubenswrapper[2578]: I0706 21:59:46.990591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:12.560112 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:00:12.560075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:12.660072 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:00:12.660038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:25.052442 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:00:25.052399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:25.271941 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:00:25.271901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:41.049496 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:00:41.049454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:57.848338 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:00:57.848299    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 22:00:57.848338 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:00:57.848300    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 22:00:58.369343 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:00:58.369307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 22:00:58.369552 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:00:58.369308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 22:01:15.834630 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:01:15.834458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:01:15.895829 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:01:15.895787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:14.668126 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:02:14.668086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:14.888998 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:02:14.888957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:38.226363 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:02:38.226329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:01.370651 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:03:01.370592    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 22:03:01.370651 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:03:01.370599    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 22:03:01.896654 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:03:01.896624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 22:03:01.896840 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:03:01.896624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 22:03:32.294967 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:03:32.294931    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:32.377441 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:03:32.377400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:08.402733 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:04:08.402677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:08.665950 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:04:08.665861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:30.463782 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:04:30.463745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:54.406620 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:04:54.406579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:54.481953 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:04:54.481916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:05:04.897890 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:05:04.897846    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 22:05:04.897890 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:05:04.897849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 22:05:05.434422 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:05.434388    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 22:05:05.434422 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:05.434427    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 22:05:15.505926 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:15.505887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:05:15.578698 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:15.578648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:05:23.132562 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:23.132512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:05:23.191916 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:23.191879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:05:32.548905 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:32.548860    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:32.557919 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:32.557891    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:34.847485 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:34.847442    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:34.861378 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:34.861331    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:37.179425 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:37.179383    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:37.191009 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:37.190981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:39.410698 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:39.410599    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:39.422737 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:39.422684    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:41.477514 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:41.477483    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:41.657073 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:41.657038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:44.056876 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:44.056836    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:44.108793 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:44.108760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:45.317200 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:45.317158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:05:46.322778 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:46.322741    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:46.332740 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:46.332696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:48.551088 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:48.551048    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:48.562611 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:48.562579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:50.773806 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:50.773767    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:50.784769 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:50.784738    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:53.045626 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:53.045594    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:53.060549 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:53.060517    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:55.277783 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:55.277747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:55.289578 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:55.289543    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:57.533114 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:57.533081    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:57.544363 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:57.544330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:05:59.741264 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:59.741228    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:05:59.752567 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:05:59.752538    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:06:01.996890 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:01.996852    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:06:02.008032 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:02.007994    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:06:04.252035 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:04.251996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:06:04.264498 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:04.264466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:06:06.516621 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:06.516588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/main/0.log"
Jul 06 22:06:06.528208 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:06.528179    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0_868a686f-eacb-42d1-8b43-c4373014e1b1/storage-initializer/0.log"
Jul 06 22:06:08.223532 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:08.223494    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/1.log"
Jul 06 22:06:08.233254 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:08.233227    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/2.log"
Jul 06 22:06:09.317854 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:09.317811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/1.log"
Jul 06 22:06:09.327494 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:09.327454    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/2.log"
Jul 06 22:06:10.410438 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:10.410404    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/1.log"
Jul 06 22:06:10.419618 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:10.419584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/2.log"
Jul 06 22:06:11.507782 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:11.507742    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/1.log"
Jul 06 22:06:11.517680 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:11.517655    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/2.log"
Jul 06 22:06:12.490378 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:12.490340    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:13.693610 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:13.693575    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:14.905555 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:14.905516    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:16.088828 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:16.088795    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:17.278567 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:17.278526    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:18.464596 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:18.464543    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:19.641464 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:19.641429    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:20.835082 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:20.835033    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:22.001099 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:22.001064    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:23.154764 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:23.154729    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:24.319059 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:24.319026    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:24.775136 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:24.775099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:06:25.467739 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:25.467681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:26.661960 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:26.661926    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:27.948797 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:27.948760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:29.183028 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:29.182994    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:30.343729 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:30.343684    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:31.503574 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:31.503536    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:32.658019 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:32.657976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:33.853042 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:33.853004    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:35.037874 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:35.037841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:36.191110 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:36.191075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:38.800984 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:38.800943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:06:38.960349 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:38.960306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:06:39.062053 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:39.061954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:06:41.982270 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:41.982235    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4jh4x_fd34120d-7c4c-418c-a18a-502b063e4a74/global-pull-secret-syncer/0.log"
Jul 06 22:06:42.106621 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:42.106578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log"
Jul 06 22:06:42.264394 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:42.264307    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-38.ec2.internal_dc314882863d165c307bf7eda7756faa/haproxy/0.log"
Jul 06 22:06:44.741236 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:44.741198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:06:44.829882 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:44.829843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22bw2_2a560bf6-4a79-4776-a235-fed8973ef3c6/konnectivity-agent/0.log" limit=1024
Jul 06 22:06:47.328844 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:47.328809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-k7wx6_0b83dded-aa86-4220-939a-e62e9b913f1e/authorino/0.log"
Jul 06 22:06:49.293403 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:49.293376    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-b2xxg_2dd0c77d-4ae2-41fd-9dd7-176f59aab38e/monitoring-plugin/0.log"
Jul 06 22:06:49.352308 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:49.352281    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f9b97_eadaa795-ad53-4038-a3c0-a53e938f4ad7/node-exporter/0.log"
Jul 06 22:06:49.381848 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:49.381817    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f9b97_eadaa795-ad53-4038-a3c0-a53e938f4ad7/kube-rbac-proxy/0.log"
Jul 06 22:06:49.408590 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:49.408564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f9b97_eadaa795-ad53-4038-a3c0-a53e938f4ad7/init-textfile/0.log"
Jul 06 22:06:52.183055 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.183019    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"]
Jul 06 22:06:52.186354 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.186333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.189809 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.189787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95ll7\"/\"openshift-service-ca.crt\""
Jul 06 22:06:52.190043 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.189804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95ll7\"/\"kube-root-ca.crt\""
Jul 06 22:06:52.190043 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.189864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-95ll7\"/\"default-dockercfg-brp8c\""
Jul 06 22:06:52.198169 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.198147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"]
Jul 06 22:06:52.239493 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.239448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-podres\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.239493 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.239502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-proc\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.239756 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.239581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-sys\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.239756 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.239630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-lib-modules\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.239756 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.239746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrg8\" (UniqueName: \"kubernetes.io/projected/c2a5f667-5596-4c93-84e2-3624b532a456-kube-api-access-dqrg8\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.340829 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-podres\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.340829 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-proc\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.341081 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-sys\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.341081 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-lib-modules\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.341081 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrg8\" (UniqueName: \"kubernetes.io/projected/c2a5f667-5596-4c93-84e2-3624b532a456-kube-api-access-dqrg8\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.341081 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340986    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-proc\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.341081 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340986    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-podres\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.341081 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.340990    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-sys\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.341081 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.341068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2a5f667-5596-4c93-84e2-3624b532a456-lib-modules\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.349899 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.349871    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrg8\" (UniqueName: \"kubernetes.io/projected/c2a5f667-5596-4c93-84e2-3624b532a456-kube-api-access-dqrg8\") pod \"perf-node-gather-daemonset-svrdr\" (UID: \"c2a5f667-5596-4c93-84e2-3624b532a456\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.496468 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.496347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.631670 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.631642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"]
Jul 06 22:06:52.634328 ip-10-0-140-38 kubenswrapper[2578]: W0706 22:06:52.634299    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc2a5f667_5596_4c93_84e2_3624b532a456.slice/crio-1530e79f29a36b52d6034a0082340409019b17f113768179345a4f11a0554a2c WatchSource:0}: Error finding container 1530e79f29a36b52d6034a0082340409019b17f113768179345a4f11a0554a2c: Status 404 returned error can't find the container with id 1530e79f29a36b52d6034a0082340409019b17f113768179345a4f11a0554a2c
Jul 06 22:06:52.901764 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.901711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr" event={"ID":"c2a5f667-5596-4c93-84e2-3624b532a456","Type":"ContainerStarted","Data":"3d62de428be761bebd4dc48b7a97f92759ba375a5ee37e30789be3ec6606a149"}
Jul 06 22:06:52.901951 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.901793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr" event={"ID":"c2a5f667-5596-4c93-84e2-3624b532a456","Type":"ContainerStarted","Data":"1530e79f29a36b52d6034a0082340409019b17f113768179345a4f11a0554a2c"}
Jul 06 22:06:52.901951 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.901826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:06:52.920017 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:52.919965    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr" podStartSLOduration=0.91994942 podStartE2EDuration="919.94942ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:06:52.918316594 +0000 UTC m=+4655.378078471" watchObservedRunningTime="2026-07-06 22:06:52.91994942 +0000 UTC m=+4655.379711296"
Jul 06 22:06:54.229701 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:54.229642    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nt7zj_8b0d5561-6a42-4921-920b-9ec87a6c4a4e/dns/0.log"
Jul 06 22:06:54.252914 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:54.252868    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nt7zj_8b0d5561-6a42-4921-920b-9ec87a6c4a4e/kube-rbac-proxy/0.log"
Jul 06 22:06:54.327991 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:54.327955    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xv2hw_8383939a-3749-4b54-b4c5-002e3df5779a/dns-node-resolver/0.log"
Jul 06 22:06:54.919843 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:54.919810    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c7f8c7bcf-nl7sm_716b91f7-c9b4-4bff-a1a7-2f89e43f8858/registry/0.log"
Jul 06 22:06:54.948417 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:54.948382    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jxf9_61661b85-2135-42c3-acb0-554ebb57828b/node-ca/0.log"
Jul 06 22:06:56.208560 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:56.208524    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/1.log"
Jul 06 22:06:56.218661 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:56.218624    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/2.log"
Jul 06 22:06:56.804044 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:56.804007    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f99wv_ab4d61a5-c075-4890-90e0-23ae50b1e952/serve-healthcheck-canary/0.log"
Jul 06 22:06:57.497035 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:57.496999    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhlj7_9c84bc98-fc5b-4b73-b610-e77a0dbda6d5/kube-rbac-proxy/0.log"
Jul 06 22:06:57.519237 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:57.519209    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhlj7_9c84bc98-fc5b-4b73-b610-e77a0dbda6d5/exporter/0.log"
Jul 06 22:06:57.542090 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:57.542055    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhlj7_9c84bc98-fc5b-4b73-b610-e77a0dbda6d5/extractor/0.log"
Jul 06 22:06:58.915621 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:06:58.915591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-svrdr"
Jul 06 22:07:00.347941 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:00.347909    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7694784d57-vvk2h_40703173-9342-41ca-ba6c-95aab8dc88d6/manager/0.log"
Jul 06 22:07:00.377051 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:00.377021    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7nlh5_cc75e51e-134d-4d69-acde-b2f17af5fabb/openshift-lws-operator/0.log"
Jul 06 22:07:02.112433 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:02.112401    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-l9wwh_92b6ed37-2176-4efb-9e65-a3b455dba9c5/seaweedfs/0.log"
Jul 06 22:07:04.026039 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:04.026008    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/1.log"
Jul 06 22:07:04.038304 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:04.038277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/2.log"
Jul 06 22:07:05.330373 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:05.330333    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/1.log"
Jul 06 22:07:05.339966 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:05.339934    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-769cdd8d88-hxzrg_07976c37-1aab-44af-ab95-9824a1a523b4/istio-proxy/2.log"
Jul 06 22:07:07.791377 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:07.791340    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-2w478_a839a4e5-8884-47ff-b9ad-67b6cfb4057a/kube-storage-version-migrator-operator/0.log"
Jul 06 22:07:08.435753 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:07:08.435697    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="678ececc-2850-4fbd-8f87-c49d5b100c52"
Jul 06 22:07:08.435753 ip-10-0-140-38 kubenswrapper[2578]: E0706 22:07:08.435697    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="793c5d9e-cfef-4986-bbf3-f36fe4151ce3"
Jul 06 22:07:08.856537 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.856510    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m4b7_1b9f1b31-50f7-47e6-940a-db563c935d88/kube-multus-additional-cni-plugins/0.log"
Jul 06 22:07:08.880095 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.880067    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m4b7_1b9f1b31-50f7-47e6-940a-db563c935d88/egress-router-binary-copy/0.log"
Jul 06 22:07:08.903204 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.903175    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m4b7_1b9f1b31-50f7-47e6-940a-db563c935d88/cni-plugins/0.log"
Jul 06 22:07:08.929529 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.929496    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m4b7_1b9f1b31-50f7-47e6-940a-db563c935d88/bond-cni-plugin/0.log"
Jul 06 22:07:08.956076 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.955985    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m4b7_1b9f1b31-50f7-47e6-940a-db563c935d88/routeoverride-cni/0.log"
Jul 06 22:07:08.975607 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.975570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 22:07:08.975801 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.975570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 22:07:08.988746 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:08.985745    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m4b7_1b9f1b31-50f7-47e6-940a-db563c935d88/whereabouts-cni-bincopy/0.log"
Jul 06 22:07:09.009884 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:09.009841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8m4b7_1b9f1b31-50f7-47e6-940a-db563c935d88/whereabouts-cni/0.log"
Jul 06 22:07:09.386140 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:09.386053    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n9hh5_bce170ed-ea9d-431e-9488-e79f29c66270/kube-multus/0.log"
Jul 06 22:07:09.443291 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:09.443254    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bg7sr_edcbaf42-d2ca-4474-ac2b-cac85ffa7805/network-metrics-daemon/0.log"
Jul 06 22:07:09.466835 ip-10-0-140-38 kubenswrapper[2578]: I0706 22:07:09.466803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bg7sr_edcbaf42-d2ca-4474-ac2b-cac85ffa7805/kube-rbac-proxy/0.log"