Jul 09 17:39:30.355352 ip-10-0-132-16 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.767054    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772895    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772907    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772911    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772915    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772918    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772921    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772924    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:30.783006 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772927    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772930    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772935    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772939    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772942    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772945    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772948    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772957    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772962    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772966    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772969    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772972    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772974    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772977    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772980    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772983    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772986    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772989    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772992    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:30.783810 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772995    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.772998    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773001    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773004    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773007    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773010    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773013    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773016    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773018    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773021    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773024    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773027    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773029    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773033    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773036    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773039    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773042    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773044    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773047    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773049    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:30.784600 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773052    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773055    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773058    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773060    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773063    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773066    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773068    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773071    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773073    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773076    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773079    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773082    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773085    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773087    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773092    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773094    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773098    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773101    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773103    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773106    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:30.789462 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773109    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773127    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773132    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773136    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773139    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773142    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773145    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773147    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773150    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773153    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773156    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773159    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773161    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773165    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773170    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773175    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773178    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773182    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773185    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773188    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:30.790373 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773191    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773194    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773196    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773199    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773202    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773204    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773208    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773210    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773213    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773216    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773218    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773221    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.773224    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776626    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776639    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776643    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776647    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776650    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776653    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:30.791350 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776656    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776659    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776662    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776664    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776667    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776670    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776673    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776676    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776678    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776704    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776708    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776712    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776715    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776718    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776722    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776725    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776728    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776731    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776734    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776736    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:30.793030 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776739    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776742    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776745    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776747    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776750    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776754    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776758    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776763    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776766    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776770    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776773    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776775    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776779    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776781    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776786    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776789    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776793    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776797    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776800    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:30.793856 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776803    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776806    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776808    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776811    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776815    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776821    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776824    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776827    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776831    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776834    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776837    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776839    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776843    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776845    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776848    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776851    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776854    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776857    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776859    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:30.794700 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776863    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776866    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776869    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776872    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776875    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776879    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776882    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776884    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776887    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776890    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776892    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776895    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776898    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776900    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776903    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776906    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776909    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776911    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776914    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776924    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:30.796018 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776927    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776930    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776933    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776935    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776938    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776941    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776944    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776947    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776949    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776952    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776955    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776958    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776961    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776963    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.776966    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777390    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777396    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777399    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777402    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777405    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:30.797068 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777411    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777414    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777417    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777419    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777422    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777425    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777428    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777430    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777433    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777436    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777439    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777443    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777446    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777448    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777451    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777454    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777457    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777460    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777462    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777465    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:30.798449 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777467    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777470    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777472    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777475    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777478    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777480    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777483    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777486    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777489    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777491    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777494    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777497    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777501    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777504    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777507    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777510    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777514    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777518    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777520    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777523    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:30.799267 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777526    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777530    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777534    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777536    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777540    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777544    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777547    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777550    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777553    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777556    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777559    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777563    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777566    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777569    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777572    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777575    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777577    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777580    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777583    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:30.800499 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777586    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777589    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777592    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777595    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777598    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777603    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777606    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777608    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777611    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777614    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777617    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777620    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777623    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777632    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777636    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777639    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777642    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777645    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777649    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:30.801777 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777652    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777655    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777661    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777664    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777667    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777669    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777672    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777675    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777677    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777680    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777683    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777685    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777688    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777691    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777693    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.777696    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777769    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777777    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777786    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777790    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777799    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 17:39:30.803572 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777803    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777807    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777811    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777815    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777819    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777822    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777826    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777829    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777832    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777836    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777839    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777842    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777845    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777850    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777853    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777856    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777859    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777863    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777867    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777870    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777873    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777877    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777880    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777883    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 17:39:30.804368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777886    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777890    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777893    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777897    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777901    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777905    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777908    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777912    2576 flags.go:64] FLAG: --enable-server="true"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777915    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777921    2576 flags.go:64] FLAG: --event-burst="100"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777924    2576 flags.go:64] FLAG: --event-qps="50"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777928    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777931    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777934    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777938    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777942    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777945    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777948    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777951    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777954    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777958    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777961    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777965    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777968    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777971    2576 flags.go:64] FLAG: --feature-gates=""
Jul 09 17:39:30.805581 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777975    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777978    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777981    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777984    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777988    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777991    2576 flags.go:64] FLAG: --help="false"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777994    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-16.ec2.internal"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.777997    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778000    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778003    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778007    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778011    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778015    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778018    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778023    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778026    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778029    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778033    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778036    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778039    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778042    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778045    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778048    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778051    2576 flags.go:64] FLAG: --lock-file=""
Jul 09 17:39:30.806421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778054    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778057    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778060    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778066    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778069    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778072    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778075    2576 flags.go:64] FLAG: --logging-format="text"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778078    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778082    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778085    2576 flags.go:64] FLAG: --manifest-url=""
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778088    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778092    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778096    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778100    2576 flags.go:64] FLAG: --max-pods="110"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778103    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778106    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778124    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778129    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778133    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778136    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778140    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778148    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778153    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778156    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 17:39:30.807670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778159    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778162    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778168    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778171    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778174    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778177    2576 flags.go:64] FLAG: --port="10250"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778181    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778184    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f57df6db85db5095"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778187    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778191    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778194    2576 flags.go:64] FLAG: --register-node="true"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778197    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778201    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778205    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778208    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778211    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778214    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778218    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778221    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778224    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778227    2576 flags.go:64] FLAG: --runonce="false"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778230    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778233    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778236    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778239    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 17:39:30.808867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778242    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778245    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778248    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778251    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778256    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778261    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778264    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778267    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778270    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778273    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778276    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778281    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778284    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778287    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778291    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778294    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778297    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778300    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778303    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778307    2576 flags.go:64] FLAG: --v="2"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778311    2576 flags.go:64] FLAG: --version="false"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778315    2576 flags.go:64] FLAG: --vmodule=""
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778319    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.778322    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 17:39:30.810964 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778455    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778459    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778462    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778465    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778468    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778470    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778474    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778477    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778480    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778483    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778485    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778489    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778495    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778499    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778502    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778506    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778509    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778512    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778515    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:30.811808 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778518    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778520    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778523    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778526    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778529    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778531    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778534    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778537    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778540    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778543    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778546    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778549    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778552    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778555    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778558    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778561    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778563    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778566    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778569    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778572    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:30.814253 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778574    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778577    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778580    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778583    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778586    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778589    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778594    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778597    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778600    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778603    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778607    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778611    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778614    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778617    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778620    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778623    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778628    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778632    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:30.815166 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778635    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778638    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778641    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778643    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778646    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778649    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778652    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778655    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778658    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778660    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778663    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778666    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778668    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778671    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778674    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778676    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778679    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778682    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778685    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778689    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778693    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:30.817026 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778695    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778698    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778701    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778704    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778706    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778709    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778712    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778715    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778717    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778720    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778723    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778726    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778729    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778732    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778734    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778737    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778740    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778742    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778749    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778752    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:30.818334 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.778755    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.779628    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.786908    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.786931    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.787911    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.787927    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788010    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788019    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788025    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788030    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788035    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788039    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788044    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788051    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:30.819206 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788056    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788061    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788066    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788071    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788075    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788081    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788085    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788090    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788095    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788100    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788105    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788110    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788130    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788134    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788138    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788144    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788148    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788152    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788156    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788161    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:30.820175 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788165    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788171    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788175    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788180    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788183    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788188    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788192    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788196    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788200    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788205    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788209    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788213    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788218    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788222    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788226    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788230    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788234    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788238    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788243    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788247    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:30.938322 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788251    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:30.851027 ip-10-0-132-16 systemd[1]: Started Kubernetes Kubelet.
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788256    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788260    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788265    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788269    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788273    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788277    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788281    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788285    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788289    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788293    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788300    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788305    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788310    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788315    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788320    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788325    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788329    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788333    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:30.939996 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788337    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788341    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788346    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788351    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788356    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788360    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788364    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788370    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788375    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788380    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788385    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788389    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788393    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788397    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788401    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788405    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788409    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788413    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788417    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:30.942085 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788421    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788426    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788430    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788434    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788438    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788443    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788447    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788451    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788456    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788461    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788464    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788468    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788472    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.788480    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788650    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:30.942942 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788656    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788661    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788666    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788671    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788675    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788680    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788684    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788689    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788693    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788698    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788703    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788707    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788711    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788716    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788720    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788725    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788729    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788733    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788737    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788741    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:30.955616 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788745    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788749    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788754    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788758    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788762    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788766    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788770    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788775    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788779    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788783    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788787    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788792    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788799    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788803    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788807    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788810    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788815    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788819    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788823    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:30.956456 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788828    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788832    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788836    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788842    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788846    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788851    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788855    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788859    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788863    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788867    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788871    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788875    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788880    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788884    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788887    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788892    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788896    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788900    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788904    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788909    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:30.958550 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788913    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788917    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788923    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788928    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788933    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788939    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788943    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788947    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788951    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788955    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788959    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788963    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788967    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788971    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788975    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788983    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788987    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788991    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788995    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:30.959402 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.788999    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789003    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789007    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789011    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789018    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789023    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789028    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789034    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789038    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789045    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789050    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789055    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789059    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789063    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789067    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789077    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789083    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789086    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789092    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:30.961505 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:30.789097    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.789105    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.789266    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.792104    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.793563    2576 server.go:1019] "Starting client certificate rotation"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.793664    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.793703    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.796154    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.797960    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.810054    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.814682    2576 log.go:25] "Validated CRI v1 image API"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.817605    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 17:39:30.962338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.821340    2576 fs.go:135] Filesystem UUIDs: map[48ed04b6-e37e-493c-b4e3-e09f550ceb56:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7f730ea8-7027-4cab-8524-dfe18b20e0ca:/dev/nvme0n1p4]
Jul 09 17:39:30.964852 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.821361    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 17:39:30.965281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.829129    2576 manager.go:216] Machine: {Timestamp:2026-07-09 17:39:30.827497385 +0000 UTC m=+0.358274409 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101714 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b2e1260553f79795ac77aa21af10 SystemUUID:ec21b2e1-2605-53f7-9795-ac77aa21af10 BootID:0e0aa3f2-c635-4165-aec4-5f0d2afdd080 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:c3:6e:e7:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:c3:6e:e7:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:80:2a:69:bd:bb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 17:39:30.965281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.829272    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 17:39:30.965281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.829404    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 17:39:30.965281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.833678    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 17:39:30.965281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.833715    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 17:39:30.965281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.834230    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:30.965281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.834435    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.834453    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.834464    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.835267    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.836601    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.839667    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.839827    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.839863    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.839875    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.839894    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.839907    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.839920    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.840333    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.840355    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.842505    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844563    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844592    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844757    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844771    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844777    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844784    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844792    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.844798    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.845484    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.845492    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.845499    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.845505    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.845515    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.845522    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.846805    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.846814    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.850054    2576 server.go:1268] "Started kubelet"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.850155    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.850318    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.850452    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.851868    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.853432    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:30.854688    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.859226    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.859235    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:30.859814    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.859920    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.859926    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.859990    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860182    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860193    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 17:39:30.968222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860494    2576 factory.go:153] Registering CRI-O factory
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860510    2576 factory.go:223] Registration of the crio container factory successfully
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860589    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860620    2576 factory.go:55] Registering systemd factory
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860629    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860654    2576 factory.go:103] Registering Raw factory
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.860669    2576 manager.go:1196] Started watching for new ooms in manager
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.861015    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:30.861624    2576 manager.go:349] Starting recovery of all containers
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:30.861931    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:30.863532    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:30.861427    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-16.ec2.internal.18c0afd2733c0f09  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-16.ec2.internal,UID:ip-10-0-132-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-16.ec2.internal,},FirstTimestamp:2026-07-09 17:39:30.850025225 +0000 UTC m=+0.380802250,LastTimestamp:2026-07-09 17:39:30.850025225 +0000 UTC m=+0.380802250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-16.ec2.internal,}"
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:30.937129    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 17:39:30.969903 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:30.937198    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 17:39:31.085745 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.084944    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:31.090403 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.086233    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fhmxl"
Jul 09 17:39:31.090403 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.086257    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fhmxl"
Jul 09 17:39:31.227098 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.223813    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:31.227098 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.223838    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Jul 09 17:39:31.232749 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.231421    2576 manager.go:354] Recovery completed
Jul 09 17:39:31.237684 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.235988    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.431125    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.431903    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.431943    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.431960    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.432555    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.432565    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.432577    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.432581    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.433386    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.433413    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.433428    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.433960    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-16.ec2.internal.18c0afd295eb18aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-16.ec2.internal,UID:ip-10-0-132-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-16.ec2.internal,},FirstTimestamp:2026-07-09 17:39:31.431921834 +0000 UTC m=+0.962698863,LastTimestamp:2026-07-09 17:39:31.431921834 +0000 UTC m=+0.962698863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-16.ec2.internal,}"
Jul 09 17:39:31.447661 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.446038    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-16.ec2.internal.18c0afd295eb89fa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-16.ec2.internal,UID:ip-10-0-132-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-16.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-16.ec2.internal,},FirstTimestamp:2026-07-09 17:39:31.431950842 +0000 UTC m=+0.962727866,LastTimestamp:2026-07-09 17:39:31.431950842 +0000 UTC m=+0.962727866,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-16.ec2.internal,}"
Jul 09 17:39:31.459743 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.459682    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-16.ec2.internal.18c0afd295ebc3f0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-16.ec2.internal,UID:ip-10-0-132-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-16.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-16.ec2.internal,},FirstTimestamp:2026-07-09 17:39:31.43196568 +0000 UTC m=+0.962742705,LastTimestamp:2026-07-09 17:39:31.43196568 +0000 UTC m=+0.962742705,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-16.ec2.internal,}"
Jul 09 17:39:31.469907 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.469835    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-16.ec2.internal.18c0afd295eb18aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-16.ec2.internal.18c0afd295eb18aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-16.ec2.internal,UID:ip-10-0-132-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-16.ec2.internal,},FirstTimestamp:2026-07-09 17:39:31.431921834 +0000 UTC m=+0.962698863,LastTimestamp:2026-07-09 17:39:31.433398236 +0000 UTC m=+0.964175262,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-16.ec2.internal,}"
Jul 09 17:39:31.480757 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.480683    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-16.ec2.internal.18c0afd295eb89fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-16.ec2.internal.18c0afd295eb89fa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-16.ec2.internal,UID:ip-10-0-132-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-16.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-16.ec2.internal,},FirstTimestamp:2026-07-09 17:39:31.431950842 +0000 UTC m=+0.962727866,LastTimestamp:2026-07-09 17:39:31.43341908 +0000 UTC m=+0.964196106,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-16.ec2.internal,}"
Jul 09 17:39:31.493742 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.493660    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-16.ec2.internal.18c0afd295ebc3f0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-16.ec2.internal.18c0afd295ebc3f0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-16.ec2.internal,UID:ip-10-0-132-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-16.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-16.ec2.internal,},FirstTimestamp:2026-07-09 17:39:31.43196568 +0000 UTC m=+0.962742705,LastTimestamp:2026-07-09 17:39:31.433433272 +0000 UTC m=+0.964210298,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-16.ec2.internal,}"
Jul 09 17:39:31.506632 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.506617    2576 policy_none.go:49] "None policy: Start"
Jul 09 17:39:31.506690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.506651    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 17:39:31.506690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.506663    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 17:39:31.507944 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.507926    2576 policy_none.go:47] "Start"
Jul 09 17:39:31.531850 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.531738    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:31.542703 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.542688    2576 manager.go:337] "Starting Device Plugin manager"
Jul 09 17:39:31.542796 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.542719    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 17:39:31.542796 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.542730    2576 server.go:87] "Starting device plugin registration server"
Jul 09 17:39:31.542975 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.542962    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 17:39:31.543009 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.542977    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 17:39:31.543078 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.543061    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 17:39:31.543215 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.543198    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 17:39:31.543215 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.543212    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 17:39:31.543789 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.543766    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 17:39:31.543875 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.543816    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:31.631883 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.631836    2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-16.ec2.internal\" not found" node="ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.643838 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.643817    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:31.645267 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.645252    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:31.645344 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.645281    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:31.645344 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.645291    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:31.645344 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.645317    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.653786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.653770    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.653879 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.653793    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-16.ec2.internal\": node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:31.661733 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.661710    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 17:39:31.662958 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.662936    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 17:39:31.663050 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.662979    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 17:39:31.663050 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.663000    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 17:39:31.663164 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.663040    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 17:39:31.665150 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.665130    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:31.668438 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.668420    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-16.ec2.internal\" not found"
Jul 09 17:39:31.753591 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.753573    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:31.764121 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.763410    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"]
Jul 09 17:39:31.766105 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.766090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.767209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.767191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.783652 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.783630    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.787336 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.787319    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.793372 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.793346    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 17:39:31.793490 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.793472    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 17:39:31.793569 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.793544    2576 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a06caf0690c83408e8e1ab76b1cacb04-6bd3147b9bc78608.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.132.16:37388->18.208.20.237:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.793617 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.793552    2576 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a06caf0690c83408e8e1ab76b1cacb04-6bd3147b9bc78608.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.132.16:37388->18.208.20.237:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.841107 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.841088    2576 apiserver.go:52] "Watching apiserver"
Jul 09 17:39:31.852954 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.852928    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 17:39:31.853319 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.853300    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pfxts","openshift-network-diagnostics/network-check-target-bhf5x","openshift-network-operator/iptables-alerter-m8wzb","kube-system/konnectivity-agent-d6l4b","kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2","openshift-dns/node-resolver-dd2pj","openshift-image-registry/node-ca-5qmgm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal","openshift-multus/multus-6qcwf","openshift-ovn-kubernetes/ovnkube-node-jwnj6","kube-system/global-pull-secret-syncer-rwpm5","openshift-cluster-node-tuning-operator/tuned-pggzv","openshift-multus/multus-additional-cni-plugins-2z6v6"]
Jul 09 17:39:31.854741 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.854727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.855773 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.855755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:31.855860 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.855823    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:31.856804 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.856788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.857419 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.857404    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 17:39:31.857562 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.857539    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 17:39:31.857764 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.857727    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 17:39:31.858039 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.858016    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s7zhl\""
Jul 09 17:39:31.858118 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.858101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:31.858182 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.858167    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 17:39:31.859286 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.859270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:31.859369 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.859315    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:31.859428 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.859395    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 17:39:31.859801 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.859785    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 17:39:31.859876 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.859821    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f4j8k\""
Jul 09 17:39:31.860439 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.860424    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.860747 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.860725    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 17:39:31.861139 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.861107    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 17:39:31.861139 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.861124    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 17:39:31.861230 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.861158    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z55rt\""
Jul 09 17:39:31.861678 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.861658    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 17:39:31.861786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.861693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:31.861786 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.861768    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:31.861896 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.861876    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 17:39:31.861945 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.861877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d2bzg\""
Jul 09 17:39:31.862905 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.862890    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 17:39:31.863056 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.863041    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 17:39:31.863206 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.863189    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 17:39:31.863273 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.863263    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cfpk7\""
Jul 09 17:39:31.863501 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.863487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:31.864771 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.864755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:31.867686 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.867527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:31.867686 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:31.867661    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:31.867921 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.867900    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:31.868052 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.868014    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn497\""
Jul 09 17:39:31.868188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.868124    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:31.868763 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.868740    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 17:39:31.869188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.868853    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 17:39:31.869188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.868873    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q565w\""
Jul 09 17:39:31.869188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.868875    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 17:39:31.869188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.868879    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 17:39:31.869188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.869031    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 17:39:31.869188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.869170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:31.869762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.869746    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 17:39:31.870462 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.870442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:31.872957 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.872941    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4cmgp\""
Jul 09 17:39:31.873258 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.873242    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 17:39:31.874196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.874177    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:31.874283 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.874200    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 17:39:31.874283 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.874249    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:31.874283 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.874261    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z2k6w\""
Jul 09 17:39:31.874437 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.874428    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 17:39:31.875471 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.875453    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:31.902959 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.902942    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz7d8"
Jul 09 17:39:31.915088 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.915057    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz7d8"
Jul 09 17:39:31.935843 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.935827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:31.935913 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.935864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.935913 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.935885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-cni-multus\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.935913 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.935902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-multus-certs\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936107 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.935927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8fn\" (UniqueName: \"kubernetes.io/projected/51a60da2-9a68-4b29-8129-086730ccfa31-kube-api-access-vv8fn\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936107 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.935975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-tmp-dir\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:31.936107 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51a60da2-9a68-4b29-8129-086730ccfa31-cni-binary-copy\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936107 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-conf-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936244 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/928a4afe-36fc-4397-9e2b-d88e2156fff4-host\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:31.936244 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-registration-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.936244 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2560b51e81baa57e27e9a93c886cb16-config\") pod \"kube-apiserver-proxy-ip-10-0-132-16.ec2.internal\" (UID: \"c2560b51e81baa57e27e9a93c886cb16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.936244 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-system-cni-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936244 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-cni-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936244 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cnibin\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/928a4afe-36fc-4397-9e2b-d88e2156fff4-serviceca\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-socket-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20d585d30a3dc2495233d9d9fffc921b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal\" (UID: \"20d585d30a3dc2495233d9d9fffc921b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-k8s-cni-cncf-io\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-cni-bin\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51a60da2-9a68-4b29-8129-086730ccfa31-multus-daemon-config\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-system-cni-dir\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-os-release\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-hosts-file\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkfd\" (UniqueName: \"kubernetes.io/projected/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-kube-api-access-rjkfd\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-cnibin\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-kubelet\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:31.936723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6whd\" (UniqueName: \"kubernetes.io/projected/928a4afe-36fc-4397-9e2b-d88e2156fff4-kube-api-access-r6whd\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9xmt\" (UniqueName: \"kubernetes.io/projected/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-kube-api-access-m9xmt\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr929\" (UniqueName: \"kubernetes.io/projected/b557bf31-5c95-4ae4-af30-2b606ec16f36-kube-api-access-qr929\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-device-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-sys-fs\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-hostroot\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-os-release\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-socket-dir-parent\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-etc-selinux\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d585d30a3dc2495233d9d9fffc921b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal\" (UID: \"20d585d30a3dc2495233d9d9fffc921b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-netns\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.937040 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.936925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-etc-kubernetes\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:31.961138 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:31.961102    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 17:39:32.038150 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysctl-conf\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.038239 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-systemd\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.038239 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-host\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.038239 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-system-cni-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.038239 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-system-cni-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-socket-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8qn\" (UniqueName: \"kubernetes.io/projected/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-kube-api-access-kt8qn\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51a60da2-9a68-4b29-8129-086730ccfa31-multus-daemon-config\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-hosts-file\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-socket-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-node-log\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-cni-bin\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.038383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-modprobe-d\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-kubelet\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-hosts-file\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6whd\" (UniqueName: \"kubernetes.io/projected/928a4afe-36fc-4397-9e2b-d88e2156fff4-kube-api-access-r6whd\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9xmt\" (UniqueName: \"kubernetes.io/projected/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-kube-api-access-m9xmt\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038499    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-kubelet\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-device-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038546    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-device-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-sys\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-var-lib-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-ovn\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovnkube-config\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.038824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovn-node-metrics-cert\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-os-release\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-socket-dir-parent\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-etc-selinux\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a87002de-8bc9-4f3c-8dad-fb393e549cf1-konnectivity-ca\") pod \"konnectivity-agent-d6l4b\" (UID: \"a87002de-8bc9-4f3c-8dad-fb393e549cf1\") " pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038837    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-os-release\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-etc-kubernetes\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-socket-dir-parent\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xptm\" (UniqueName: \"kubernetes.io/projected/269e8c04-7a45-4450-8394-d4c231425a77-kube-api-access-8xptm\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-etc-selinux\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51a60da2-9a68-4b29-8129-086730ccfa31-multus-daemon-config\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9t5f\" (UniqueName: \"kubernetes.io/projected/b115ad4b-87b6-4463-a9a7-72054517de4f-kube-api-access-c9t5f\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038929    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-etc-kubernetes\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.039353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-cni-netd\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.038987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-host-slash\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-cni-multus\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-multus-certs\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-cni-multus\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-kubernetes\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-var-lib-kubelet\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-multus-certs\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51a60da2-9a68-4b29-8129-086730ccfa31-cni-binary-copy\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-slash\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twd9g\" (UniqueName: \"kubernetes.io/projected/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-kube-api-access-twd9g\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a87002de-8bc9-4f3c-8dad-fb393e549cf1-agent-certs\") pod \"konnectivity-agent-d6l4b\" (UID: \"a87002de-8bc9-4f3c-8dad-fb393e549cf1\") " pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:32.040017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2560b51e81baa57e27e9a93c886cb16-config\") pod \"kube-apiserver-proxy-ip-10-0-132-16.ec2.internal\" (UID: \"c2560b51e81baa57e27e9a93c886cb16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-cni-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cnibin\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/928a4afe-36fc-4397-9e2b-d88e2156fff4-serviceca\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20d585d30a3dc2495233d9d9fffc921b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal\" (UID: \"20d585d30a3dc2495233d9d9fffc921b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039353    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2560b51e81baa57e27e9a93c886cb16-config\") pod \"kube-apiserver-proxy-ip-10-0-132-16.ec2.internal\" (UID: \"c2560b51e81baa57e27e9a93c886cb16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cnibin\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-cni-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20d585d30a3dc2495233d9d9fffc921b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal\" (UID: \"20d585d30a3dc2495233d9d9fffc921b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-systemd\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-k8s-cni-cncf-io\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-cni-bin\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-system-cni-dir\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.040762 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039550    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-k8s-cni-cncf-io\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-os-release\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-var-lib-cni-bin\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039608    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-system-cni-dir\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysconfig\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-kubelet\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039675    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51a60da2-9a68-4b29-8129-086730ccfa31-cni-binary-copy\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-iptables-alerter-script\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039660    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-os-release\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkfd\" (UniqueName: \"kubernetes.io/projected/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-kube-api-access-rjkfd\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/928a4afe-36fc-4397-9e2b-d88e2156fff4-serviceca\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-cnibin\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-cnibin\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr929\" (UniqueName: \"kubernetes.io/projected/b557bf31-5c95-4ae4-af30-2b606ec16f36-kube-api-access-qr929\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-sys-fs\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-etc-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/01ffe74e-2c33-429c-8818-fb8d7fa9f969-kubelet-config\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-hostroot\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-sys-fs\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.039982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d585d30a3dc2495233d9d9fffc921b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal\" (UID: \"20d585d30a3dc2495233d9d9fffc921b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-run-netns\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040071    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-hostroot\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d585d30a3dc2495233d9d9fffc921b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal\" (UID: \"20d585d30a3dc2495233d9d9fffc921b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040098    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.041608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-log-socket\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-netns\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040214    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-host-run-netns\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040223    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-lib-modules\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-tuned\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b115ad4b-87b6-4463-a9a7-72054517de4f-tmp\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovnkube-script-lib\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8fn\" (UniqueName: \"kubernetes.io/projected/51a60da2-9a68-4b29-8129-086730ccfa31-kube-api-access-vv8fn\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-tmp-dir\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-run\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-systemd-units\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-env-overrides\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.042044 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/01ffe74e-2c33-429c-8818-fb8d7fa9f969-dbus\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-conf-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/928a4afe-36fc-4397-9e2b-d88e2156fff4-host\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/928a4afe-36fc-4397-9e2b-d88e2156fff4-host\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51a60da2-9a68-4b29-8129-086730ccfa31-multus-conf-dir\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-registration-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040616    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b557bf31-5c95-4ae4-af30-2b606ec16f36-registration-dir\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysctl-d\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.042455 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.040665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-tmp-dir\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:32.051612 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.051595    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 17:39:32.052568 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.052551    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:32.052568 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.052572    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:32.052702 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.052582    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wlnl7 for pod openshift-network-diagnostics/network-check-target-bhf5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:32.052702 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.052646    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7 podName:99c84757-5b96-4311-a959-0936ed20dd67 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:32.552626327 +0000 UTC m=+2.083403341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wlnl7" (UniqueName: "kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7") pod "network-check-target-bhf5x" (UID: "99c84757-5b96-4311-a959-0936ed20dd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:32.055068 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.055046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8fn\" (UniqueName: \"kubernetes.io/projected/51a60da2-9a68-4b29-8129-086730ccfa31-kube-api-access-vv8fn\") pod \"multus-6qcwf\" (UID: \"51a60da2-9a68-4b29-8129-086730ccfa31\") " pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.055213 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.055126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr929\" (UniqueName: \"kubernetes.io/projected/b557bf31-5c95-4ae4-af30-2b606ec16f36-kube-api-access-qr929\") pod \"aws-ebs-csi-driver-node-mq2h2\" (UID: \"b557bf31-5c95-4ae4-af30-2b606ec16f36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.055539 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.055515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6whd\" (UniqueName: \"kubernetes.io/projected/928a4afe-36fc-4397-9e2b-d88e2156fff4-kube-api-access-r6whd\") pod \"node-ca-5qmgm\" (UID: \"928a4afe-36fc-4397-9e2b-d88e2156fff4\") " pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:32.056177 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.056155    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkfd\" (UniqueName: \"kubernetes.io/projected/ac0fe359-dcb4-4dbc-842f-eae3a47ddac7-kube-api-access-rjkfd\") pod \"multus-additional-cni-plugins-2z6v6\" (UID: \"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7\") " pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.056529 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.056514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9xmt\" (UniqueName: \"kubernetes.io/projected/6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4-kube-api-access-m9xmt\") pod \"node-resolver-dd2pj\" (UID: \"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4\") " pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:32.096122 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.096088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.096639 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.096623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal"
Jul 09 17:39:32.141344 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xptm\" (UniqueName: \"kubernetes.io/projected/269e8c04-7a45-4450-8394-d4c231425a77-kube-api-access-8xptm\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:32.141344 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9t5f\" (UniqueName: \"kubernetes.io/projected/b115ad4b-87b6-4463-a9a7-72054517de4f-kube-api-access-c9t5f\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-cni-netd\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-host-slash\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-kubernetes\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-var-lib-kubelet\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-host-slash\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-cni-netd\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-slash\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-var-lib-kubelet\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.141554 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-kubernetes\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141587    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-run-ovn-kubernetes\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-slash\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.141603    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twd9g\" (UniqueName: \"kubernetes.io/projected/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-kube-api-access-twd9g\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a87002de-8bc9-4f3c-8dad-fb393e549cf1-agent-certs\") pod \"konnectivity-agent-d6l4b\" (UID: \"a87002de-8bc9-4f3c-8dad-fb393e549cf1\") " pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.141662    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret podName:01ffe74e-2c33-429c-8818-fb8d7fa9f969 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:32.641646571 +0000 UTC m=+2.172423605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret") pod "global-pull-secret-syncer-rwpm5" (UID: "01ffe74e-2c33-429c-8818-fb8d7fa9f969") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-systemd\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-systemd\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysconfig\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysconfig\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.142030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-kubelet\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-iptables-alerter-script\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-kubelet\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-etc-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.141987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/01ffe74e-2c33-429c-8818-fb8d7fa9f969-kubelet-config\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-run-netns\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-etc-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-log-socket\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142073    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-run-netns\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142090    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/01ffe74e-2c33-429c-8818-fb8d7fa9f969-kubelet-config\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-lib-modules\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-log-socket\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-tuned\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.142786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b115ad4b-87b6-4463-a9a7-72054517de4f-tmp\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovnkube-script-lib\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-run\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-systemd-units\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-env-overrides\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-lib-modules\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/01ffe74e-2c33-429c-8818-fb8d7fa9f969-dbus\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysctl-d\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-run\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysctl-conf\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/01ffe74e-2c33-429c-8818-fb8d7fa9f969-dbus\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142463    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-iptables-alerter-script\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142521    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysctl-d\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-systemd-units\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-systemd\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143448 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-systemd\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-host\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8qn\" (UniqueName: \"kubernetes.io/projected/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-kube-api-access-kt8qn\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-sysctl-conf\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-node-log\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-cni-bin\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142685    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-host\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-modprobe-d\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-sys\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142767    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-host-cni-bin\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-var-lib-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-ovn\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovnkube-config\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.142853    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:32.143979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovn-node-metrics-cert\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a87002de-8bc9-4f3c-8dad-fb393e549cf1-konnectivity-ca\") pod \"konnectivity-agent-d6l4b\" (UID: \"a87002de-8bc9-4f3c-8dad-fb393e549cf1\") " pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.142903    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs podName:269e8c04-7a45-4450-8394-d4c231425a77 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:32.642886676 +0000 UTC m=+2.173663688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs") pod "network-metrics-daemon-pfxts" (UID: "269e8c04-7a45-4450-8394-d4c231425a77") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142917    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-var-lib-openvswitch\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142733    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-node-log\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142941    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-env-overrides\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.142980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-sys\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.143024    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-modprobe-d\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.143032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovnkube-script-lib\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.143081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-run-ovn\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.143497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a87002de-8bc9-4f3c-8dad-fb393e549cf1-konnectivity-ca\") pod \"konnectivity-agent-d6l4b\" (UID: \"a87002de-8bc9-4f3c-8dad-fb393e549cf1\") " pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:32.144610 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.143530    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovnkube-config\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.145022 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.144720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a87002de-8bc9-4f3c-8dad-fb393e549cf1-agent-certs\") pod \"konnectivity-agent-d6l4b\" (UID: \"a87002de-8bc9-4f3c-8dad-fb393e549cf1\") " pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:32.145022 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.144752    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b115ad4b-87b6-4463-a9a7-72054517de4f-etc-tuned\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.145137 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.145105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b115ad4b-87b6-4463-a9a7-72054517de4f-tmp\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.145424 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.145402    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-ovn-node-metrics-cert\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.150921 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.150856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9t5f\" (UniqueName: \"kubernetes.io/projected/b115ad4b-87b6-4463-a9a7-72054517de4f-kube-api-access-c9t5f\") pod \"tuned-pggzv\" (UID: \"b115ad4b-87b6-4463-a9a7-72054517de4f\") " pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.151406 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.151382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8qn\" (UniqueName: \"kubernetes.io/projected/ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41-kube-api-access-kt8qn\") pod \"iptables-alerter-m8wzb\" (UID: \"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41\") " pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.151723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.151706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twd9g\" (UniqueName: \"kubernetes.io/projected/dd4e8426-56fe-4848-9f7a-7d00b5108aa7-kube-api-access-twd9g\") pod \"ovnkube-node-jwnj6\" (UID: \"dd4e8426-56fe-4848-9f7a-7d00b5108aa7\") " pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.152488 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.152470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xptm\" (UniqueName: \"kubernetes.io/projected/269e8c04-7a45-4450-8394-d4c231425a77-kube-api-access-8xptm\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:32.177867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.177847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6qcwf"
Jul 09 17:39:32.196504 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.196487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2z6v6"
Jul 09 17:39:32.201189 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.201170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5qmgm"
Jul 09 17:39:32.224981 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.224963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dd2pj"
Jul 09 17:39:32.240666 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.240642    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2"
Jul 09 17:39:32.263001 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.262979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pggzv"
Jul 09 17:39:32.270164 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.270146    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:32.271548 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.271529    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:32.283576 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.283560    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:32.287179 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.287164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m8wzb"
Jul 09 17:39:32.422206 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.422143    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:32.508535 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.508480    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:34:31 +0000 UTC" deadline="2028-01-21 02:39:11.255586745 +0000 UTC"
Jul 09 17:39:32.508535 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.508528    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13448h59m38.747060957s"
Jul 09 17:39:32.646297 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.646254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.646324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.646363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646446    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646499    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646451    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646519    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646531    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wlnl7 for pod openshift-network-diagnostics/network-check-target-bhf5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646533    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret podName:01ffe74e-2c33-429c-8818-fb8d7fa9f969 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:33.646511648 +0000 UTC m=+3.177288681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret") pod "global-pull-secret-syncer-rwpm5" (UID: "01ffe74e-2c33-429c-8818-fb8d7fa9f969") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646577    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs podName:269e8c04-7a45-4450-8394-d4c231425a77 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:33.646555591 +0000 UTC m=+3.177332621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs") pod "network-metrics-daemon-pfxts" (UID: "269e8c04-7a45-4450-8394-d4c231425a77") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:32.646594 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.646593    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7 podName:99c84757-5b96-4311-a959-0936ed20dd67 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:33.64658552 +0000 UTC m=+3.177362531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlnl7" (UniqueName: "kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7") pod "network-check-target-bhf5x" (UID: "99c84757-5b96-4311-a959-0936ed20dd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:32.674188 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.674141    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:32.679244 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.679220    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0fe359_dcb4_4dbc_842f_eae3a47ddac7.slice/crio-0bad65062df5f7ded747b7dcb022e73235ad79a0b79ac70cb4f65723b9517c3b WatchSource:0}: Error finding container 0bad65062df5f7ded747b7dcb022e73235ad79a0b79ac70cb4f65723b9517c3b: Status 404 returned error can't find the container with id 0bad65062df5f7ded747b7dcb022e73235ad79a0b79ac70cb4f65723b9517c3b
Jul 09 17:39:32.679693 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.679680    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6107f0d9_9d5a_4e33_b248_eb8b3ad63fe4.slice/crio-88fb2a4746e06ad1dad8e2ebad1f7dd4e98d9c3b55c71c943833470f47eff3d4 WatchSource:0}: Error finding container 88fb2a4746e06ad1dad8e2ebad1f7dd4e98d9c3b55c71c943833470f47eff3d4: Status 404 returned error can't find the container with id 88fb2a4746e06ad1dad8e2ebad1f7dd4e98d9c3b55c71c943833470f47eff3d4
Jul 09 17:39:32.685576 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.685482    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:39:32.699660 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.699641    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a60da2_9a68_4b29_8129_086730ccfa31.slice/crio-56ee479ee10713d49fc0e0131cd2a0a3ca026afae94f36e5f3ec170b50bdd00c WatchSource:0}: Error finding container 56ee479ee10713d49fc0e0131cd2a0a3ca026afae94f36e5f3ec170b50bdd00c: Status 404 returned error can't find the container with id 56ee479ee10713d49fc0e0131cd2a0a3ca026afae94f36e5f3ec170b50bdd00c
Jul 09 17:39:32.720380 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.720361    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2560b51e81baa57e27e9a93c886cb16.slice/crio-cd298ec90c2918bc493685de3f094108b7030905139f2ea63e682a80a317f017 WatchSource:0}: Error finding container cd298ec90c2918bc493685de3f094108b7030905139f2ea63e682a80a317f017: Status 404 returned error can't find the container with id cd298ec90c2918bc493685de3f094108b7030905139f2ea63e682a80a317f017
Jul 09 17:39:32.740269 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.740248    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4e8426_56fe_4848_9f7a_7d00b5108aa7.slice/crio-47c6cac49221e75e7115ff5177e6f738f3cdec32134c6aed65276f42e6b8c6db WatchSource:0}: Error finding container 47c6cac49221e75e7115ff5177e6f738f3cdec32134c6aed65276f42e6b8c6db: Status 404 returned error can't find the container with id 47c6cac49221e75e7115ff5177e6f738f3cdec32134c6aed65276f42e6b8c6db
Jul 09 17:39:32.742320 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.742292    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2d2bc6_e7c5_42f0_a1f2_7c99a6a68c41.slice/crio-715842dfafe4ad58237c5263600fe5dac25fa3aafa76ebb6461aef4f11b2ab67 WatchSource:0}: Error finding container 715842dfafe4ad58237c5263600fe5dac25fa3aafa76ebb6461aef4f11b2ab67: Status 404 returned error can't find the container with id 715842dfafe4ad58237c5263600fe5dac25fa3aafa76ebb6461aef4f11b2ab67
Jul 09 17:39:32.755201 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.755180    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928a4afe_36fc_4397_9e2b_d88e2156fff4.slice/crio-00a0749dd2aa804a17c4e2e85e84f21e36c00e3823df2c6121b0f7ef06621770 WatchSource:0}: Error finding container 00a0749dd2aa804a17c4e2e85e84f21e36c00e3823df2c6121b0f7ef06621770: Status 404 returned error can't find the container with id 00a0749dd2aa804a17c4e2e85e84f21e36c00e3823df2c6121b0f7ef06621770
Jul 09 17:39:32.755797 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.755769    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87002de_8bc9_4f3c_8dad_fb393e549cf1.slice/crio-a90d72ab50039ba6a8f4b412912baeb83108fdfba44ff6a6582f430be7b40cec WatchSource:0}: Error finding container a90d72ab50039ba6a8f4b412912baeb83108fdfba44ff6a6582f430be7b40cec: Status 404 returned error can't find the container with id a90d72ab50039ba6a8f4b412912baeb83108fdfba44ff6a6582f430be7b40cec
Jul 09 17:39:32.779225 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.779205    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d585d30a3dc2495233d9d9fffc921b.slice/crio-ba3372bf9476e037f99d85546609a1a562e718aca08f0b278a46fe7a9babaff7 WatchSource:0}: Error finding container ba3372bf9476e037f99d85546609a1a562e718aca08f0b278a46fe7a9babaff7: Status 404 returned error can't find the container with id ba3372bf9476e037f99d85546609a1a562e718aca08f0b278a46fe7a9babaff7
Jul 09 17:39:32.855513 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.855489    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb557bf31_5c95_4ae4_af30_2b606ec16f36.slice/crio-f9555a5ef3f6631854854c731b3f94e932f1de4454bc72e18a8364dcad75916c WatchSource:0}: Error finding container f9555a5ef3f6631854854c731b3f94e932f1de4454bc72e18a8364dcad75916c: Status 404 returned error can't find the container with id f9555a5ef3f6631854854c731b3f94e932f1de4454bc72e18a8364dcad75916c
Jul 09 17:39:32.861442 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:39:32.861422    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb115ad4b_87b6_4463_a9a7_72054517de4f.slice/crio-53b8cfdceea63f175d4b18419437c3895897d615d4b0e6b6c3ca0f0f6ee924ed WatchSource:0}: Error finding container 53b8cfdceea63f175d4b18419437c3895897d615d4b0e6b6c3ca0f0f6ee924ed: Status 404 returned error can't find the container with id 53b8cfdceea63f175d4b18419437c3895897d615d4b0e6b6c3ca0f0f6ee924ed
Jul 09 17:39:32.863034 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.863013    2576 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-pggzv_openshift-cluster-node-tuning-operator(b115ad4b-87b6-4463-a9a7-72054517de4f): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 09 17:39:32.863108 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:32.863051    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-pggzv" podUID="b115ad4b-87b6-4463-a9a7-72054517de4f"
Jul 09 17:39:32.917115 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.917065    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:32 +0000 UTC" deadline="2028-02-21 11:02:12.765720001 +0000 UTC"
Jul 09 17:39:32.917197 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:32.917129    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14201h22m39.848614668s"
Jul 09 17:39:33.654990 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.654955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:33.655214 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.655023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:33.655214 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.655079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:33.655214 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655209    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:33.655472 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655279    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret podName:01ffe74e-2c33-429c-8818-fb8d7fa9f969 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.655258854 +0000 UTC m=+5.186035885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret") pod "global-pull-secret-syncer-rwpm5" (UID: "01ffe74e-2c33-429c-8818-fb8d7fa9f969") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:33.655733 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655710    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:33.655814 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655744    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:33.655814 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655758    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wlnl7 for pod openshift-network-diagnostics/network-check-target-bhf5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:33.655814 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655806    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7 podName:99c84757-5b96-4311-a959-0936ed20dd67 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.655790245 +0000 UTC m=+5.186567259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlnl7" (UniqueName: "kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7") pod "network-check-target-bhf5x" (UID: "99c84757-5b96-4311-a959-0936ed20dd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:33.655986 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655873    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:33.655986 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.655910    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs podName:269e8c04-7a45-4450-8394-d4c231425a77 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.655898505 +0000 UTC m=+5.186675537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs") pod "network-metrics-daemon-pfxts" (UID: "269e8c04-7a45-4450-8394-d4c231425a77") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:33.664725 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.664666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:33.664854 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.664670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:33.664854 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.664794    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:33.664987 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.664896    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:33.665056 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.664984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:33.665164 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.665135    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:33.693247 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.693192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dd2pj" event={"ID":"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4","Type":"ContainerStarted","Data":"88fb2a4746e06ad1dad8e2ebad1f7dd4e98d9c3b55c71c943833470f47eff3d4"}
Jul 09 17:39:33.709250 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.709179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerStarted","Data":"0bad65062df5f7ded747b7dcb022e73235ad79a0b79ac70cb4f65723b9517c3b"}
Jul 09 17:39:33.721108 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.720800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pggzv" event={"ID":"b115ad4b-87b6-4463-a9a7-72054517de4f","Type":"ContainerStarted","Data":"53b8cfdceea63f175d4b18419437c3895897d615d4b0e6b6c3ca0f0f6ee924ed"}
Jul 09 17:39:33.725227 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:33.725001    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-pggzv" podUID="b115ad4b-87b6-4463-a9a7-72054517de4f"
Jul 09 17:39:33.754090 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.754045    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d6l4b" event={"ID":"a87002de-8bc9-4f3c-8dad-fb393e549cf1","Type":"ContainerStarted","Data":"a90d72ab50039ba6a8f4b412912baeb83108fdfba44ff6a6582f430be7b40cec"}
Jul 09 17:39:33.771751 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.771719    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5qmgm" event={"ID":"928a4afe-36fc-4397-9e2b-d88e2156fff4","Type":"ContainerStarted","Data":"00a0749dd2aa804a17c4e2e85e84f21e36c00e3823df2c6121b0f7ef06621770"}
Jul 09 17:39:33.801659 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.801629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m8wzb" event={"ID":"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41","Type":"ContainerStarted","Data":"715842dfafe4ad58237c5263600fe5dac25fa3aafa76ebb6461aef4f11b2ab67"}
Jul 09 17:39:33.808897 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.808868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal" event={"ID":"c2560b51e81baa57e27e9a93c886cb16","Type":"ContainerStarted","Data":"cd298ec90c2918bc493685de3f094108b7030905139f2ea63e682a80a317f017"}
Jul 09 17:39:33.825633 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.825606    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2" event={"ID":"b557bf31-5c95-4ae4-af30-2b606ec16f36","Type":"ContainerStarted","Data":"f9555a5ef3f6631854854c731b3f94e932f1de4454bc72e18a8364dcad75916c"}
Jul 09 17:39:33.839654 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.839620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal" event={"ID":"20d585d30a3dc2495233d9d9fffc921b","Type":"ContainerStarted","Data":"ba3372bf9476e037f99d85546609a1a562e718aca08f0b278a46fe7a9babaff7"}
Jul 09 17:39:33.852526 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.852493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"47c6cac49221e75e7115ff5177e6f738f3cdec32134c6aed65276f42e6b8c6db"}
Jul 09 17:39:33.861752 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.861728    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6qcwf" event={"ID":"51a60da2-9a68-4b29-8129-086730ccfa31","Type":"ContainerStarted","Data":"56ee479ee10713d49fc0e0131cd2a0a3ca026afae94f36e5f3ec170b50bdd00c"}
Jul 09 17:39:33.917564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.917491    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:32 +0000 UTC" deadline="2027-12-12 19:56:27.17901936 +0000 UTC"
Jul 09 17:39:33.917564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:33.917527    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12506h16m53.261498254s"
Jul 09 17:39:35.665189 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:35.665161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:35.665627 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:35.665166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:35.665627 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.665297    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:35.665627 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.665400    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:35.665627 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:35.665466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:35.665627 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.665535    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:35.678324 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:35.678244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:35.678324 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:35.678297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:35.678499 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:35.678345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:35.678556 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678497    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:35.678556 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678518    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:35.678556 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678529    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wlnl7 for pod openshift-network-diagnostics/network-check-target-bhf5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:35.678695 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678577    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7 podName:99c84757-5b96-4311-a959-0936ed20dd67 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:39.678558679 +0000 UTC m=+9.209335690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlnl7" (UniqueName: "kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7") pod "network-check-target-bhf5x" (UID: "99c84757-5b96-4311-a959-0936ed20dd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:35.678695 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678645    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:35.678695 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678678    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs podName:269e8c04-7a45-4450-8394-d4c231425a77 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:39.678665803 +0000 UTC m=+9.209442815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs") pod "network-metrics-daemon-pfxts" (UID: "269e8c04-7a45-4450-8394-d4c231425a77") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:35.678859 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678728    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:35.678859 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:35.678758    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret podName:01ffe74e-2c33-429c-8818-fb8d7fa9f969 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:39.678747989 +0000 UTC m=+9.209525001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret") pod "global-pull-secret-syncer-rwpm5" (UID: "01ffe74e-2c33-429c-8818-fb8d7fa9f969") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:37.663403 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:37.663371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:37.663952 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:37.663371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:37.663952 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:37.663514    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:37.663952 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:37.663628    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:37.663952 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:37.663643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:37.663952 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:37.663756    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:39.664161 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:39.664132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:39.664161 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:39.664147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:39.664662 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:39.664174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:39.664662 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.664268    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:39.664662 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.664351    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:39.664662 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.664440    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:39.711022 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:39.710992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:39.711196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:39.711064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:39.711196 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:39.711105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:39.711309 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711250    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:39.711309 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711274    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:39.711309 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711274    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:39.711309 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711288    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wlnl7 for pod openshift-network-diagnostics/network-check-target-bhf5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:39.711309 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711287    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:39.711538 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711331    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret podName:01ffe74e-2c33-429c-8818-fb8d7fa9f969 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:47.711311474 +0000 UTC m=+17.242088484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret") pod "global-pull-secret-syncer-rwpm5" (UID: "01ffe74e-2c33-429c-8818-fb8d7fa9f969") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:39.711538 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711361    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7 podName:99c84757-5b96-4311-a959-0936ed20dd67 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:47.711351204 +0000 UTC m=+17.242128214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlnl7" (UniqueName: "kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7") pod "network-check-target-bhf5x" (UID: "99c84757-5b96-4311-a959-0936ed20dd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:39.711538 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:39.711382    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs podName:269e8c04-7a45-4450-8394-d4c231425a77 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:47.711367052 +0000 UTC m=+17.242144066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs") pod "network-metrics-daemon-pfxts" (UID: "269e8c04-7a45-4450-8394-d4c231425a77") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:41.664528 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:41.664495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:41.664950 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:41.664582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:41.664950 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:41.664596    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:41.664950 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:41.664701    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:41.664950 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:41.664735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:41.664950 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:41.664812    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:43.663364 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:43.663318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:43.663815 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:43.663322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:43.663815 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:43.663461    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:43.663815 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:43.663330    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:43.663815 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:43.663546    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:43.664164 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:43.664139    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:45.663450 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:45.663428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:45.663741 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:45.663432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:45.663741 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:45.663516    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:45.663741 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:45.663586    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:45.663741 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:45.663627    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:45.663741 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:45.663692    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:47.663771 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:47.663732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:47.663771 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:47.663754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:47.664302 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.663879    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:47.664302 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.663942    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:47.664302 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:47.664005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:47.664302 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.664104    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:47.772564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:47.772524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:47.772726 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:47.772591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:47.772726 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772622    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:47.772726 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:47.772638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:47.772726 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772678    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret podName:01ffe74e-2c33-429c-8818-fb8d7fa9f969 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:03.772658839 +0000 UTC m=+33.303435871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret") pod "global-pull-secret-syncer-rwpm5" (UID: "01ffe74e-2c33-429c-8818-fb8d7fa9f969") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:47.772726 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772716    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:47.772960 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772752    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs podName:269e8c04-7a45-4450-8394-d4c231425a77 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:03.77274087 +0000 UTC m=+33.303517892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs") pod "network-metrics-daemon-pfxts" (UID: "269e8c04-7a45-4450-8394-d4c231425a77") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:47.772960 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772822    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:47.772960 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772852    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:47.772960 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772863    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wlnl7 for pod openshift-network-diagnostics/network-check-target-bhf5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:47.772960 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:47.772896    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7 podName:99c84757-5b96-4311-a959-0936ed20dd67 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:03.772885452 +0000 UTC m=+33.303662475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlnl7" (UniqueName: "kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7") pod "network-check-target-bhf5x" (UID: "99c84757-5b96-4311-a959-0936ed20dd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:49.663740 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:49.663679    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:49.664194 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:49.663693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:49.664194 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:49.663808    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:49.664194 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:49.663846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:49.664194 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:49.663962    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:49.664194 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:49.664033    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:50.908297 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.907932    2576 generic.go:356] "Generic (PLEG): container finished" podID="20d585d30a3dc2495233d9d9fffc921b" containerID="3f3c8b7dd530376f6a1f56d2b1a2776bb33be06c08721b160dd29ccc77570433" exitCode=0
Jul 09 17:39:50.909126 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.908023    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal" event={"ID":"20d585d30a3dc2495233d9d9fffc921b","Type":"ContainerDied","Data":"3f3c8b7dd530376f6a1f56d2b1a2776bb33be06c08721b160dd29ccc77570433"}
Jul 09 17:39:50.911726 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.911699    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:39:50.912162 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.912131    2576 generic.go:356] "Generic (PLEG): container finished" podID="dd4e8426-56fe-4848-9f7a-7d00b5108aa7" containerID="e2526168285ff6bae098dd9ef4f5e75a232b6c65ca1d6ab553f03eb156679e72" exitCode=1
Jul 09 17:39:50.912281 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.912204    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"4ac49ecac84910c5460bd6766fe9841b145f8a93ab27f7578d8655dfc7cae1d0"}
Jul 09 17:39:50.912359 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.912280    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"8a2e8156b4aacb321f10789e30ea9e1b2335047de94686f7b739bb198791b9eb"}
Jul 09 17:39:50.912359 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.912330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"1a9f987fac661178ffe152a5844e9ada8ed700ccb0a6e419113024e9aaf9f695"}
Jul 09 17:39:50.912472 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.912372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"b1417c76bd15d092ad1c5697b12a1167ca544a8b52436c47709e5425af7210bf"}
Jul 09 17:39:50.912472 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.912419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerDied","Data":"e2526168285ff6bae098dd9ef4f5e75a232b6c65ca1d6ab553f03eb156679e72"}
Jul 09 17:39:50.912472 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.912459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"34acf69bde95c0eefa8b772ff19aec9561d23f3f073f29745e5ecf380c98177b"}
Jul 09 17:39:50.913840 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.913793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6qcwf" event={"ID":"51a60da2-9a68-4b29-8129-086730ccfa31","Type":"ContainerStarted","Data":"8080298b402ce03cf1788e4ac4af97a8b94802075560bdc67091f0a15430ecff"}
Jul 09 17:39:50.915423 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.915404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dd2pj" event={"ID":"6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4","Type":"ContainerStarted","Data":"6aeb3b9f62a2fd8e20a5bdf481616964eead91da6cfe9e68c931e73dcc62da3d"}
Jul 09 17:39:50.917179 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.917132    2576 generic.go:356] "Generic (PLEG): container finished" podID="ac0fe359-dcb4-4dbc-842f-eae3a47ddac7" containerID="ad3e44bcabab447fb4f54cbac9ecb05bb717af9881470dc621d0f6bb08a802ae" exitCode=0
Jul 09 17:39:50.917280 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.917214    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerDied","Data":"ad3e44bcabab447fb4f54cbac9ecb05bb717af9881470dc621d0f6bb08a802ae"}
Jul 09 17:39:50.919561 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.919257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d6l4b" event={"ID":"a87002de-8bc9-4f3c-8dad-fb393e549cf1","Type":"ContainerStarted","Data":"6104d683ee39ba7e5d9e754b4630109b4f50bce2b403ae6bc02b448e723e2bae"}
Jul 09 17:39:50.920838 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.920815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5qmgm" event={"ID":"928a4afe-36fc-4397-9e2b-d88e2156fff4","Type":"ContainerStarted","Data":"3d54e88f4147471565392ba46b3f83c621072b7800589d2b88b54e7dedaa5888"}
Jul 09 17:39:50.922392 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.922362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal" event={"ID":"c2560b51e81baa57e27e9a93c886cb16","Type":"ContainerStarted","Data":"45f4b8fc005510c0a93394fc046cabe392dc10e23c68a779bb58fbcd5c60f196"}
Jul 09 17:39:50.924057 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.924021    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2" event={"ID":"b557bf31-5c95-4ae4-af30-2b606ec16f36","Type":"ContainerStarted","Data":"633a0c6c308814bdd6c35f6a7e9f37bb08caf8bc42ca8c8e088d1acd791cd576"}
Jul 09 17:39:50.950447 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.950380    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d6l4b" podStartSLOduration=1.682502356 podStartE2EDuration="18.950366557s" totalImagesPullingTime="17.267864201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:50.950123045 +0000 UTC m=+20.480900076" watchObservedRunningTime="2026-07-09 17:39:50.950366557 +0000 UTC m=+20.481143615"
Jul 09 17:39:50.966487 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.966450    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal" podStartSLOduration=18.966436649 podStartE2EDuration="18.966436649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:50.965219902 +0000 UTC m=+20.495996933" watchObservedRunningTime="2026-07-09 17:39:50.966436649 +0000 UTC m=+20.497213679"
Jul 09 17:39:50.983642 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:50.983598    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5qmgm" podStartSLOduration=2.71509172 podStartE2EDuration="19.983581118s" totalImagesPullingTime="17.268489398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:50.982798679 +0000 UTC m=+20.513575710" watchObservedRunningTime="2026-07-09 17:39:50.983581118 +0000 UTC m=+20.514358149"
Jul 09 17:39:51.026899 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.026697    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dd2pj" podStartSLOduration=2.686928178 podStartE2EDuration="20.02667925s" totalImagesPullingTime="17.339751072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:51.003346242 +0000 UTC m=+20.534123273" watchObservedRunningTime="2026-07-09 17:39:51.02667925 +0000 UTC m=+20.557456281"
Jul 09 17:39:51.027358 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.027074    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6qcwf" podStartSLOduration=2.646885754 podStartE2EDuration="20.027059467s" totalImagesPullingTime="17.380173713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:51.026609126 +0000 UTC m=+20.557386163" watchObservedRunningTime="2026-07-09 17:39:51.027059467 +0000 UTC m=+20.557836497"
Jul 09 17:39:51.566926 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.566864    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 17:39:51.665210 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.665180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:51.665358 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:51.665283    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:51.665622 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.665601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:51.665721 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.665709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:51.665781 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:51.665719    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:51.665828 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:51.665792    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:51.929116 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.929080    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m8wzb" event={"ID":"ca2d2bc6-e7c5-42f0-a1f2-7c99a6a68c41","Type":"ContainerStarted","Data":"287e050720886eb8abc9d8783599b4c9443448f58ef2e72bd8f2678767efd4ea"}
Jul 09 17:39:51.931572 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.931506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2" event={"ID":"b557bf31-5c95-4ae4-af30-2b606ec16f36","Type":"ContainerStarted","Data":"ac58357236d4f07c4e1b9426fdb6aa45da9da9a5c968ddf3a538c765971b858a"}
Jul 09 17:39:51.934079 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.933942    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal" event={"ID":"20d585d30a3dc2495233d9d9fffc921b","Type":"ContainerStarted","Data":"36a017daa289f97a396cb5e5eb281a9236024e6f1d47442cacc6ee717e79d798"}
Jul 09 17:39:51.943915 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.943868    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m8wzb" podStartSLOduration=2.626444449 podStartE2EDuration="19.943850961s" totalImagesPullingTime="17.317406512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:51.943705196 +0000 UTC m=+21.474482239" watchObservedRunningTime="2026-07-09 17:39:51.943850961 +0000 UTC m=+21.474627994"
Jul 09 17:39:51.960083 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:51.960022    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-16.ec2.internal" podStartSLOduration=19.960009669 podStartE2EDuration="19.960009669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:51.959823439 +0000 UTC m=+21.490600471" watchObservedRunningTime="2026-07-09 17:39:51.960009669 +0000 UTC m=+21.490786701"
Jul 09 17:39:52.554481 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:52.554322    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T17:39:51.566884214Z","UUID":"2e151989-236d-4896-8199-6650a8078680","Handler":null,"Name":"","Endpoint":""}
Jul 09 17:39:52.555991 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:52.555966    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 17:39:52.556117 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:52.556006    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 17:39:53.664162 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:53.664131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:53.664725 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:53.664135    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:53.664725 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:53.664267    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:53.664725 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:53.664281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:53.664725 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:53.664382    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:53.664725 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:53.664471    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:54.094889 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:54.094861    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:54.095432 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:54.095415    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:54.939874 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:54.939842    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:54.940391 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:54.940325    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d6l4b"
Jul 09 17:39:55.663881 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:55.663847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:55.664073 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:55.663847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:55.664073 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:55.663975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:55.664073 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:55.663970    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:55.664239 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:55.664103    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:55.664239 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:55.664188    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:56.948557 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:56.948530    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:39:56.949237 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:56.949212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"4e894e3476c0f22d5a7795eadb0c9961d5c0fbab38c136e491e6699111dd7c7b"}
Jul 09 17:39:57.663802 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.663645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:57.663939 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.663654    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:57.663982 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:57.663947    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:57.663982 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:57.663865    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:57.663982 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.663669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:57.664126 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:57.664008    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:57.952258 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.952189    2576 generic.go:356] "Generic (PLEG): container finished" podID="ac0fe359-dcb4-4dbc-842f-eae3a47ddac7" containerID="98e4617a417f792a488cf50b24e33b8fe9175ffff940fa4c9bae7215666ed295" exitCode=0
Jul 09 17:39:57.952617 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.952275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerDied","Data":"98e4617a417f792a488cf50b24e33b8fe9175ffff940fa4c9bae7215666ed295"}
Jul 09 17:39:57.953823 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.953805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pggzv" event={"ID":"b115ad4b-87b6-4463-a9a7-72054517de4f","Type":"ContainerStarted","Data":"25bf3676d06a98b287a114647d5ea6b7b3d73e518ce5259f70847163cf7e5694"}
Jul 09 17:39:57.955801 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.955773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2" event={"ID":"b557bf31-5c95-4ae4-af30-2b606ec16f36","Type":"ContainerStarted","Data":"2d27c143f53f917f4a3a9f2fa2af90fa1e923636097e586578ddc1bd232a2b73"}
Jul 09 17:39:57.996733 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:57.996695    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2h2" podStartSLOduration=3.618133014 podStartE2EDuration="26.996683904s" totalImagesPullingTime="23.37855089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:57.996490981 +0000 UTC m=+27.527268012" watchObservedRunningTime="2026-07-09 17:39:57.996683904 +0000 UTC m=+27.527460934"
Jul 09 17:39:58.017383 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:58.017351    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pggzv" podStartSLOduration=3.057978597 podStartE2EDuration="27.017340307s" totalImagesPullingTime="23.95936171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 17:39:58.017137904 +0000 UTC m=+27.547914924" watchObservedRunningTime="2026-07-09 17:39:58.017340307 +0000 UTC m=+27.548117352"
Jul 09 17:39:58.959836 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:58.959778    2576 generic.go:356] "Generic (PLEG): container finished" podID="ac0fe359-dcb4-4dbc-842f-eae3a47ddac7" containerID="12e175b2cf19a1c8f203938c7fae8d78afa51ef2feed770c644bc0f0ada0419d" exitCode=0
Jul 09 17:39:58.960409 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:58.959860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerDied","Data":"12e175b2cf19a1c8f203938c7fae8d78afa51ef2feed770c644bc0f0ada0419d"}
Jul 09 17:39:59.664184 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.664017    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:39:59.664303 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.664022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:39:59.664303 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:59.664287    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:39:59.664406 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.664031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:39:59.664406 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:59.664327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:39:59.664493 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:39:59.664412    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:39:59.964445 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.964424    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:39:59.964937 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.964725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"813f8b42fb80315dee1a013ccc317d39b4086a2c3b61a4afa3b2547dd593c1db"}
Jul 09 17:39:59.964983 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.964955    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:59.965031 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.965014    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:59.965102 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.965081    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:59.965150 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.965128    2576 scope.go:117] "RemoveContainer" containerID="e2526168285ff6bae098dd9ef4f5e75a232b6c65ca1d6ab553f03eb156679e72"
Jul 09 17:39:59.967339 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.967313    2576 generic.go:356] "Generic (PLEG): container finished" podID="ac0fe359-dcb4-4dbc-842f-eae3a47ddac7" containerID="aceec0978b99864db08a9b5ecf79ddc6de15f62752b1f104d4a595657e01e2c3" exitCode=0
Jul 09 17:39:59.967422 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.967348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerDied","Data":"aceec0978b99864db08a9b5ecf79ddc6de15f62752b1f104d4a595657e01e2c3"}
Jul 09 17:39:59.982636 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.982609    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:39:59.982755 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:39:59.982741    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:40:00.974349 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:00.974317    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:40:00.974802 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:00.974736    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" event={"ID":"dd4e8426-56fe-4848-9f7a-7d00b5108aa7","Type":"ContainerStarted","Data":"d06c08fdffa31ddec8425756ccb4228ba13bdcf0a8df2dc915a04ec88c08bd3a"}
Jul 09 17:40:01.013867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:01.013823    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6" podStartSLOduration=11.520526181 podStartE2EDuration="29.013804032s" totalImagesPullingTime="17.493277851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:01.012291476 +0000 UTC m=+30.543068549" watchObservedRunningTime="2026-07-09 17:40:01.013804032 +0000 UTC m=+30.544581063"
Jul 09 17:40:01.443713 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:01.443673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bhf5x"]
Jul 09 17:40:01.443878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:01.443836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:01.443960 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:01.443935    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:01.446283 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:01.446254    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rwpm5"]
Jul 09 17:40:01.446407 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:01.446361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:01.446465 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:01.446445    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:01.446881 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:01.446859    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pfxts"]
Jul 09 17:40:01.447011 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:01.446954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:01.447097 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:01.447023    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:02.663399 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:02.663373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:02.663887 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:02.663373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:02.663887 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:02.663482    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:02.663887 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:02.663554    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:02.663887 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:02.663373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:02.663887 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:02.663691    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:03.800067 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:03.800016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:03.800081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:03.800127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800172    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800241    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800258    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800270    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wlnl7 for pod openshift-network-diagnostics/network-check-target-bhf5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800242    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs podName:269e8c04-7a45-4450-8394-d4c231425a77 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:35.800223761 +0000 UTC m=+65.331000778 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs") pod "network-metrics-daemon-pfxts" (UID: "269e8c04-7a45-4450-8394-d4c231425a77") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800305    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800313    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7 podName:99c84757-5b96-4311-a959-0936ed20dd67 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:35.800303085 +0000 UTC m=+65.331080094 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlnl7" (UniqueName: "kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7") pod "network-check-target-bhf5x" (UID: "99c84757-5b96-4311-a959-0936ed20dd67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:03.800520 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:03.800395    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret podName:01ffe74e-2c33-429c-8818-fb8d7fa9f969 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:35.8003769 +0000 UTC m=+65.331153911 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret") pod "global-pull-secret-syncer-rwpm5" (UID: "01ffe74e-2c33-429c-8818-fb8d7fa9f969") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:04.664005 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:04.663799    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:04.664131 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:04.663830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:04.664131 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:04.664066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:04.664192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:04.663862    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:04.664233 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:04.664136    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:04.664264 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:04.664222    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:04.985993 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:04.985921    2576 generic.go:356] "Generic (PLEG): container finished" podID="ac0fe359-dcb4-4dbc-842f-eae3a47ddac7" containerID="a15ef5a817a6f89044ec4668482ff7ced30460347ba08cd984d154509719b2fc" exitCode=0
Jul 09 17:40:04.985993 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:04.985964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerDied","Data":"a15ef5a817a6f89044ec4668482ff7ced30460347ba08cd984d154509719b2fc"}
Jul 09 17:40:05.993903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:05.993869    2576 generic.go:356] "Generic (PLEG): container finished" podID="ac0fe359-dcb4-4dbc-842f-eae3a47ddac7" containerID="5b88c22cb2a74ec5e8d6711138207bed336107387c2c0c251018c9799a8c7030" exitCode=0
Jul 09 17:40:05.993903 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:05.993907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerDied","Data":"5b88c22cb2a74ec5e8d6711138207bed336107387c2c0c251018c9799a8c7030"}
Jul 09 17:40:06.663278 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:06.663247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:06.663475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:06.663320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:06.663475 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:06.663336    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:06.663475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:06.663346    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:06.663475 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:06.663406    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:06.663620 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:06.663484    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:06.999855 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:06.999786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" event={"ID":"ac0fe359-dcb4-4dbc-842f-eae3a47ddac7","Type":"ContainerStarted","Data":"81a03e4d05c5df26b573485f8391e764bbab53d0372cfda8cf3881d51394079e"}
Jul 09 17:40:07.036359 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:07.036319    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2z6v6" podStartSLOduration=7.210197216 podStartE2EDuration="36.036304001s" totalImagesPullingTime="28.826106785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:07.035863416 +0000 UTC m=+36.566640470" watchObservedRunningTime="2026-07-09 17:40:07.036304001 +0000 UTC m=+36.567081030"
Jul 09 17:40:08.046665 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:08.046642    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dd2pj_6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4/dns-node-resolver/0.log"
Jul 09 17:40:08.596805 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:08.596780    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5qmgm_928a4afe-36fc-4397-9e2b-d88e2156fff4/node-ca/0.log"
Jul 09 17:40:08.664186 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:08.664162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:08.664310 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:08.664240    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:08.664310 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:08.664249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:08.664310 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:08.664244    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:08.664432 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:08.664326    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:08.664432 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:08.664405    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:10.663461 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:10.663428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:10.663804 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:10.663470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:10.663804 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:10.663521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:10.663804 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:10.663537    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:10.663804 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:10.663580    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:10.663804 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:10.663661    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:12.663676 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:12.663509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:12.664085 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:12.663520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:12.664085 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:12.663745    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:12.664085 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:12.663854    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:12.664085 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:12.663532    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:12.664085 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:12.663951    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:14.663395 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:14.663370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:14.663786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:14.663373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:14.663786 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:14.663373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:14.663786 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:14.663464    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:14.663786 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:14.663600    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:14.663786 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:14.663669    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:16.664058 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:16.664018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:16.664490 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:16.664113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:16.664490 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:16.664120    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:16.664490 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:16.664184    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:16.664490 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:16.664210    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:16.664490 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:16.664354    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:18.663251 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:18.663224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:18.663626 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:18.663325    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:18.663626 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:18.663351    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:18.663626 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:18.663390    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:18.663626 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:18.663428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:18.663626 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:18.663485    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:20.664228 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:20.664196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:20.664613 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:20.664296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:20.664613 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:20.664311    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:20.664613 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:20.664301    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:20.664613 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:20.664373    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:20.664613 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:20.664465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:22.663355 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:22.663327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:22.663803 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:22.663420    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:22.663803 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:22.663444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:22.663803 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:22.663434    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:22.663803 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:22.663504    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:22.663803 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:22.663564    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:24.663291 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:24.663260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:24.663291 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:24.663281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:24.663820 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:24.663260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:24.663820 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:24.663375    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:24.663820 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:24.663425    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:24.663820 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:24.663483    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:26.663608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:26.663577    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:26.663608 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:26.663598    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:26.664017 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:26.663577    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:26.664017 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:26.663671    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:26.664017 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:26.663767    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:26.664017 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:26.663822    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:28.664089 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:28.664057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:28.664589 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:28.664151    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhf5x" podUID="99c84757-5b96-4311-a959-0936ed20dd67"
Jul 09 17:40:28.664589 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:28.664058    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:28.664589 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:28.664178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:28.664589 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:28.664228    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rwpm5" podUID="01ffe74e-2c33-429c-8818-fb8d7fa9f969"
Jul 09 17:40:28.664589 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:28.664293    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfxts" podUID="269e8c04-7a45-4450-8394-d4c231425a77"
Jul 09 17:40:29.129809 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.129786    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-16.ec2.internal" event="NodeReady"
Jul 09 17:40:29.129936 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.129870    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 17:40:29.171316 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.171292    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"]
Jul 09 17:40:29.194592 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.194571    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dzsz9"]
Jul 09 17:40:29.194697 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.194687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.197751 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.197732    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 17:40:29.197893 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.197855    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 17:40:29.198016 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.197998    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 17:40:29.198110 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.198015    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5d6lq\""
Jul 09 17:40:29.205460 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.205439    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 17:40:29.211052 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.211019    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-znkdl"]
Jul 09 17:40:29.211146 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.211024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.213850 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.213830    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 17:40:29.213932 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.213882    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dxrqg\""
Jul 09 17:40:29.213986 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.213937    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 17:40:29.223198 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.223173    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"]
Jul 09 17:40:29.223198 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.223196    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-znkdl"]
Jul 09 17:40:29.223323 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.223209    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dzsz9"]
Jul 09 17:40:29.223323 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.223230    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fsgn4"]
Jul 09 17:40:29.223323 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.223293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.227213 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.227197    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2srhc\""
Jul 09 17:40:29.227292 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.227233    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 17:40:29.227420 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.227405    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 17:40:29.227482 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.227441    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 17:40:29.240994 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.240976    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fsgn4"]
Jul 09 17:40:29.241084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.241073    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.243438 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.243422    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 17:40:29.243694 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.243676    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 17:40:29.243996 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.243978    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 17:40:29.244101 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.243994    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 17:40:29.244186 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.244151    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2r4b\""
Jul 09 17:40:29.388192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388146    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5tsj\" (UniqueName: \"kubernetes.io/projected/1aeeb33f-1695-44b2-a531-6430ee5619de-kube-api-access-f5tsj\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.388192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82621485-6679-4d1a-9732-09d4db44ce8b-registry-certificates\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.388192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-bound-sa-token\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.388360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1aeeb33f-1695-44b2-a531-6430ee5619de-metrics-tls\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.388360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf5c6f12-38fd-4951-bf72-18aecc136c1f-data-volume\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.388360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388278    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130ee9fd-9318-4595-843a-484b2747d946-cert\") pod \"ingress-canary-znkdl\" (UID: \"130ee9fd-9318-4595-843a-484b2747d946\") " pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.388360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82621485-6679-4d1a-9732-09d4db44ce8b-image-registry-private-configuration\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.388360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82621485-6679-4d1a-9732-09d4db44ce8b-ca-trust-extracted\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.388360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeeb33f-1695-44b2-a531-6430ee5619de-tmp-dir\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.388360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcw6\" (UniqueName: \"kubernetes.io/projected/bf5c6f12-38fd-4951-bf72-18aecc136c1f-kube-api-access-wbcw6\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82621485-6679-4d1a-9732-09d4db44ce8b-trusted-ca\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1aeeb33f-1695-44b2-a531-6430ee5619de-config-volume\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7j25\" (UniqueName: \"kubernetes.io/projected/130ee9fd-9318-4595-843a-484b2747d946-kube-api-access-p7j25\") pod \"ingress-canary-znkdl\" (UID: \"130ee9fd-9318-4595-843a-484b2747d946\") " pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7qk\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-kube-api-access-sg7qk\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5c6f12-38fd-4951-bf72-18aecc136c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82621485-6679-4d1a-9732-09d4db44ce8b-installation-pull-secrets\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf5c6f12-38fd-4951-bf72-18aecc136c1f-crio-socket\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5c6f12-38fd-4951-bf72-18aecc136c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.388657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.388620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-registry-tls\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.489470 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130ee9fd-9318-4595-843a-484b2747d946-cert\") pod \"ingress-canary-znkdl\" (UID: \"130ee9fd-9318-4595-843a-484b2747d946\") " pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.489571 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82621485-6679-4d1a-9732-09d4db44ce8b-image-registry-private-configuration\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.489571 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82621485-6679-4d1a-9732-09d4db44ce8b-ca-trust-extracted\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.489571 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeeb33f-1695-44b2-a531-6430ee5619de-tmp-dir\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.489715 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcw6\" (UniqueName: \"kubernetes.io/projected/bf5c6f12-38fd-4951-bf72-18aecc136c1f-kube-api-access-wbcw6\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.489715 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82621485-6679-4d1a-9732-09d4db44ce8b-trusted-ca\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.489715 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1aeeb33f-1695-44b2-a531-6430ee5619de-config-volume\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.489715 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7j25\" (UniqueName: \"kubernetes.io/projected/130ee9fd-9318-4595-843a-484b2747d946-kube-api-access-p7j25\") pod \"ingress-canary-znkdl\" (UID: \"130ee9fd-9318-4595-843a-484b2747d946\") " pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.489895 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7qk\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-kube-api-access-sg7qk\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.489895 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489756    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aeeb33f-1695-44b2-a531-6430ee5619de-tmp-dir\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.489895 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5c6f12-38fd-4951-bf72-18aecc136c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.489895 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82621485-6679-4d1a-9732-09d4db44ce8b-installation-pull-secrets\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.489895 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf5c6f12-38fd-4951-bf72-18aecc136c1f-crio-socket\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.489895 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82621485-6679-4d1a-9732-09d4db44ce8b-ca-trust-extracted\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.490250 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf5c6f12-38fd-4951-bf72-18aecc136c1f-crio-socket\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.490250 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.489876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5c6f12-38fd-4951-bf72-18aecc136c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.490250 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-registry-tls\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.490250 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5tsj\" (UniqueName: \"kubernetes.io/projected/1aeeb33f-1695-44b2-a531-6430ee5619de-kube-api-access-f5tsj\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.490413 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82621485-6679-4d1a-9732-09d4db44ce8b-registry-certificates\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.490413 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-bound-sa-token\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.490413 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1aeeb33f-1695-44b2-a531-6430ee5619de-metrics-tls\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.490413 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf5c6f12-38fd-4951-bf72-18aecc136c1f-data-volume\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.490621 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490602    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf5c6f12-38fd-4951-bf72-18aecc136c1f-data-volume\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.490621 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5c6f12-38fd-4951-bf72-18aecc136c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.490705 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1aeeb33f-1695-44b2-a531-6430ee5619de-config-volume\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.490747 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.490714    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82621485-6679-4d1a-9732-09d4db44ce8b-trusted-ca\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.491361 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.491319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82621485-6679-4d1a-9732-09d4db44ce8b-registry-certificates\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.493884 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.493859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1aeeb33f-1695-44b2-a531-6430ee5619de-metrics-tls\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.493960 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.493883    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82621485-6679-4d1a-9732-09d4db44ce8b-installation-pull-secrets\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.493960 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.493920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5c6f12-38fd-4951-bf72-18aecc136c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.494025 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.493964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130ee9fd-9318-4595-843a-484b2747d946-cert\") pod \"ingress-canary-znkdl\" (UID: \"130ee9fd-9318-4595-843a-484b2747d946\") " pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.494115 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.494098    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-registry-tls\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.494452 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.494432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82621485-6679-4d1a-9732-09d4db44ce8b-image-registry-private-configuration\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.500178 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.500152    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcw6\" (UniqueName: \"kubernetes.io/projected/bf5c6f12-38fd-4951-bf72-18aecc136c1f-kube-api-access-wbcw6\") pod \"insights-runtime-extractor-fsgn4\" (UID: \"bf5c6f12-38fd-4951-bf72-18aecc136c1f\") " pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.501782 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.501754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7j25\" (UniqueName: \"kubernetes.io/projected/130ee9fd-9318-4595-843a-484b2747d946-kube-api-access-p7j25\") pod \"ingress-canary-znkdl\" (UID: \"130ee9fd-9318-4595-843a-484b2747d946\") " pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.502212 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.502157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5tsj\" (UniqueName: \"kubernetes.io/projected/1aeeb33f-1695-44b2-a531-6430ee5619de-kube-api-access-f5tsj\") pod \"dns-default-dzsz9\" (UID: \"1aeeb33f-1695-44b2-a531-6430ee5619de\") " pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.502471 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.502451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-bound-sa-token\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.503150 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.503115    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7qk\" (UniqueName: \"kubernetes.io/projected/82621485-6679-4d1a-9732-09d4db44ce8b-kube-api-access-sg7qk\") pod \"image-registry-7dbc88fcb8-4rjtv\" (UID: \"82621485-6679-4d1a-9732-09d4db44ce8b\") " pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.503255 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.503223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:29.521852 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.521832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:29.530785 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.530764    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-znkdl"
Jul 09 17:40:29.548751 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.548729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fsgn4"
Jul 09 17:40:29.699758 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.699737    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"]
Jul 09 17:40:29.702444 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:29.702421    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82621485_6679_4d1a_9732_09d4db44ce8b.slice/crio-bd04c76614048084717d40865d6a29ebaa2e8a945da5c67fb18411cc5a6b4164 WatchSource:0}: Error finding container bd04c76614048084717d40865d6a29ebaa2e8a945da5c67fb18411cc5a6b4164: Status 404 returned error can't find the container with id bd04c76614048084717d40865d6a29ebaa2e8a945da5c67fb18411cc5a6b4164
Jul 09 17:40:29.733824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.733801    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fsgn4"]
Jul 09 17:40:29.736620 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:29.736583    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5c6f12_38fd_4951_bf72_18aecc136c1f.slice/crio-ec6c4d5afaef93075d0dd8970a3ac1fb1c616752897a6fa9c0e5b6d9548e0f36 WatchSource:0}: Error finding container ec6c4d5afaef93075d0dd8970a3ac1fb1c616752897a6fa9c0e5b6d9548e0f36: Status 404 returned error can't find the container with id ec6c4d5afaef93075d0dd8970a3ac1fb1c616752897a6fa9c0e5b6d9548e0f36
Jul 09 17:40:29.910939 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.910862    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-znkdl"]
Jul 09 17:40:29.911595 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:29.911563    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dzsz9"]
Jul 09 17:40:29.913441 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:29.913372    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130ee9fd_9318_4595_843a_484b2747d946.slice/crio-e8a6c380d13fde67914c4bcb6d416f771646b00603b8df57c59f9933fc5a6a34 WatchSource:0}: Error finding container e8a6c380d13fde67914c4bcb6d416f771646b00603b8df57c59f9933fc5a6a34: Status 404 returned error can't find the container with id e8a6c380d13fde67914c4bcb6d416f771646b00603b8df57c59f9933fc5a6a34
Jul 09 17:40:29.915248 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:29.915222    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aeeb33f_1695_44b2_a531_6430ee5619de.slice/crio-34194f6fd26034b2b1747466e54e507541f1f7833e76e815e2f64f67e2336321 WatchSource:0}: Error finding container 34194f6fd26034b2b1747466e54e507541f1f7833e76e815e2f64f67e2336321: Status 404 returned error can't find the container with id 34194f6fd26034b2b1747466e54e507541f1f7833e76e815e2f64f67e2336321
Jul 09 17:40:30.046148 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.046054    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsgn4" event={"ID":"bf5c6f12-38fd-4951-bf72-18aecc136c1f","Type":"ContainerStarted","Data":"ec6c4d5afaef93075d0dd8970a3ac1fb1c616752897a6fa9c0e5b6d9548e0f36"}
Jul 09 17:40:30.047443 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.047415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-znkdl" event={"ID":"130ee9fd-9318-4595-843a-484b2747d946","Type":"ContainerStarted","Data":"e8a6c380d13fde67914c4bcb6d416f771646b00603b8df57c59f9933fc5a6a34"}
Jul 09 17:40:30.048629 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.048592    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzsz9" event={"ID":"1aeeb33f-1695-44b2-a531-6430ee5619de","Type":"ContainerStarted","Data":"34194f6fd26034b2b1747466e54e507541f1f7833e76e815e2f64f67e2336321"}
Jul 09 17:40:30.049960 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.049940    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv" event={"ID":"82621485-6679-4d1a-9732-09d4db44ce8b","Type":"ContainerStarted","Data":"c4acb9d8e505025f1bcd05f94a3206ff5f62d21955697b12524d96afd53bc616"}
Jul 09 17:40:30.050090 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.049977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv" event={"ID":"82621485-6679-4d1a-9732-09d4db44ce8b","Type":"ContainerStarted","Data":"bd04c76614048084717d40865d6a29ebaa2e8a945da5c67fb18411cc5a6b4164"}
Jul 09 17:40:30.050157 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.050143    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:30.664056 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.664002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:30.664475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.664457    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:30.664821 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.664803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:30.679557 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.678426    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 17:40:30.679557 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.678655    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 17:40:30.679557 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.678887    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wznvh\""
Jul 09 17:40:30.679557 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.679109    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 17:40:30.679557 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.679355    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 17:40:30.679557 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:30.679459    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d4pd\""
Jul 09 17:40:31.988459 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:31.988427    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jwnj6"
Jul 09 17:40:32.021400 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:32.021338    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv" podStartSLOduration=4.021318296 podStartE2EDuration="4.021318296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:30.073025231 +0000 UTC m=+59.603802289" watchObservedRunningTime="2026-07-09 17:40:32.021318296 +0000 UTC m=+61.552095327"
Jul 09 17:40:33.416183 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.416128    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5698fc6bc8-tsvw9"]
Jul 09 17:40:33.418999 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.418978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.423640 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.423619    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 17:40:33.424086 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.424033    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 17:40:33.424086 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.424061    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 17:40:33.424086 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.424066    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 17:40:33.425210 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.425186    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 17:40:33.425287 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.425218    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 17:40:33.425287 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.425260    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-76l56\""
Jul 09 17:40:33.425490 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.425477    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 17:40:33.445079 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.445052    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5698fc6bc8-tsvw9"]
Jul 09 17:40:33.520839 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.520808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-oauth-config\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.520961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.520852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-oauth-serving-cert\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.520961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.520919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-serving-cert\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.520961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.520952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-console-config\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.521123 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.520979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbhm\" (UniqueName: \"kubernetes.io/projected/97cbc575-2150-49bf-a52d-49f3877d81a0-kube-api-access-wqbhm\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.521175 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.521119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-service-ca\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.622275 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.622242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-service-ca\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.622431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.622295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-oauth-config\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.622431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.622332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-oauth-serving-cert\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.622431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.622356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-serving-cert\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.622431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.622371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-console-config\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.622431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.622393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbhm\" (UniqueName: \"kubernetes.io/projected/97cbc575-2150-49bf-a52d-49f3877d81a0-kube-api-access-wqbhm\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.623193 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.623145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-service-ca\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.623402 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.623354    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-oauth-serving-cert\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.623402 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.623396    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-console-config\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.625270 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.625243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-serving-cert\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.625373 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.625243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-oauth-config\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.636475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.636453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbhm\" (UniqueName: \"kubernetes.io/projected/97cbc575-2150-49bf-a52d-49f3877d81a0-kube-api-access-wqbhm\") pod \"console-5698fc6bc8-tsvw9\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") " pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:33.729499 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:33.729437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:34.844566 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:34.844491    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5698fc6bc8-tsvw9"]
Jul 09 17:40:34.850266 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:34.850211    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cbc575_2150_49bf_a52d_49f3877d81a0.slice/crio-6de7e3b9ac16181b4c0b9a9de0e5d53cfb70f7c1a7f2d9dbcec9cbe84a2dfcce WatchSource:0}: Error finding container 6de7e3b9ac16181b4c0b9a9de0e5d53cfb70f7c1a7f2d9dbcec9cbe84a2dfcce: Status 404 returned error can't find the container with id 6de7e3b9ac16181b4c0b9a9de0e5d53cfb70f7c1a7f2d9dbcec9cbe84a2dfcce
Jul 09 17:40:35.056068 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.055984    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"]
Jul 09 17:40:35.062670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.062649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"
Jul 09 17:40:35.066458 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.066438    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 09 17:40:35.066582 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.066516    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-62dk4\""
Jul 09 17:40:35.066835 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.066812    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsgn4" event={"ID":"bf5c6f12-38fd-4951-bf72-18aecc136c1f","Type":"ContainerStarted","Data":"4d98e1f954462887e7f39d69e9eba0792bd911cec09f8ae4e128962c8930760e"}
Jul 09 17:40:35.068260 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.068239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-znkdl" event={"ID":"130ee9fd-9318-4595-843a-484b2747d946","Type":"ContainerStarted","Data":"46fc62f0cb6293661e65821f0fa48281e9d84403ecdd19d9cdc4eb617d9023b9"}
Jul 09 17:40:35.071293 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.071263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzsz9" event={"ID":"1aeeb33f-1695-44b2-a531-6430ee5619de","Type":"ContainerStarted","Data":"d7b3a3e50aabaadd49e5a6bd1b1323fde490d7d613f90b1194e9a88774fcf5b0"}
Jul 09 17:40:35.071408 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.071355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzsz9" event={"ID":"1aeeb33f-1695-44b2-a531-6430ee5619de","Type":"ContainerStarted","Data":"4da8c77a118553ed7d65b07b924b7e7d32c86a618aa923a6672ea99046f3b965"}
Jul 09 17:40:35.071510 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.071492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:35.071772 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.071754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"]
Jul 09 17:40:35.076310 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.076280    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698fc6bc8-tsvw9" event={"ID":"97cbc575-2150-49bf-a52d-49f3877d81a0","Type":"ContainerStarted","Data":"6de7e3b9ac16181b4c0b9a9de0e5d53cfb70f7c1a7f2d9dbcec9cbe84a2dfcce"}
Jul 09 17:40:35.120176 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.120139    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dzsz9" podStartSLOduration=1.3411360860000001 podStartE2EDuration="6.120126689s" totalImagesPullingTime="4.778990603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:35.120094809 +0000 UTC m=+64.650871838" watchObservedRunningTime="2026-07-09 17:40:35.120126689 +0000 UTC m=+64.650903719"
Jul 09 17:40:35.133675 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.133651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/03d952ac-152f-427f-85cd-9672adcc63db-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-khplm\" (UID: \"03d952ac-152f-427f-85cd-9672adcc63db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"
Jul 09 17:40:35.149048 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.149008    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-znkdl" podStartSLOduration=1.368733036 podStartE2EDuration="6.148997781s" totalImagesPullingTime="4.780264745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:35.148328841 +0000 UTC m=+64.679105876" watchObservedRunningTime="2026-07-09 17:40:35.148997781 +0000 UTC m=+64.679774812"
Jul 09 17:40:35.234660 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.234634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/03d952ac-152f-427f-85cd-9672adcc63db-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-khplm\" (UID: \"03d952ac-152f-427f-85cd-9672adcc63db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"
Jul 09 17:40:35.236804 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.236787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/03d952ac-152f-427f-85cd-9672adcc63db-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-khplm\" (UID: \"03d952ac-152f-427f-85cd-9672adcc63db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"
Jul 09 17:40:35.377092 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.377072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"
Jul 09 17:40:35.553577 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.553548    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"]
Jul 09 17:40:35.557406 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:35.557377    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d952ac_152f_427f_85cd_9672adcc63db.slice/crio-24a6d0cd338b98f410d9a743793f230660ab308d4a62fc40a8a4d97587102865 WatchSource:0}: Error finding container 24a6d0cd338b98f410d9a743793f230660ab308d4a62fc40a8a4d97587102865: Status 404 returned error can't find the container with id 24a6d0cd338b98f410d9a743793f230660ab308d4a62fc40a8a4d97587102865
Jul 09 17:40:35.780752 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.780687    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bf9fc9d64-zx59g"]
Jul 09 17:40:35.793858 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.793832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.796565 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.796465    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf9fc9d64-zx59g"]
Jul 09 17:40:35.803008 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.802987    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 17:40:35.841120 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-oauth-serving-cert\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.841227 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bt7\" (UniqueName: \"kubernetes.io/projected/8ee041d1-d800-4605-92e2-4c1a91d7e487-kube-api-access-z2bt7\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.841227 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-oauth-config\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.841308 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-config\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.841347 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:35.841347 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-serving-cert\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.841442 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:35.841442 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-trusted-ca-bundle\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.841442 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-service-ca\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.841585 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.841461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:35.844156 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.844060    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 17:40:35.844156 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.844090    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 17:40:35.844156 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.844073    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 17:40:35.854394 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.854372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01ffe74e-2c33-429c-8818-fb8d7fa9f969-original-pull-secret\") pod \"global-pull-secret-syncer-rwpm5\" (UID: \"01ffe74e-2c33-429c-8818-fb8d7fa9f969\") " pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:35.854701 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.854489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269e8c04-7a45-4450-8394-d4c231425a77-metrics-certs\") pod \"network-metrics-daemon-pfxts\" (UID: \"269e8c04-7a45-4450-8394-d4c231425a77\") " pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:35.854701 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.854561    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 17:40:35.864377 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.864355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnl7\" (UniqueName: \"kubernetes.io/projected/99c84757-5b96-4311-a959-0936ed20dd67-kube-api-access-wlnl7\") pod \"network-check-target-bhf5x\" (UID: \"99c84757-5b96-4311-a959-0936ed20dd67\") " pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:35.942487 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.942460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-serving-cert\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.942604 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.942531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-trusted-ca-bundle\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.942604 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.942560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-service-ca\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.942604 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.942587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-oauth-serving-cert\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.942759 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.942605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bt7\" (UniqueName: \"kubernetes.io/projected/8ee041d1-d800-4605-92e2-4c1a91d7e487-kube-api-access-z2bt7\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.942759 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.942625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-oauth-config\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.942759 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.942667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-config\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.943462 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.943395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-oauth-serving-cert\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.943462 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.943407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-service-ca\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.943462 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.943431    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-config\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.943707 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.943602    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-trusted-ca-bundle\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.945231 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.945208    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-serving-cert\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.945324 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.945301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-oauth-config\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:35.952192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:35.952161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bt7\" (UniqueName: \"kubernetes.io/projected/8ee041d1-d800-4605-92e2-4c1a91d7e487-kube-api-access-z2bt7\") pod \"console-bf9fc9d64-zx59g\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") " pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:36.082642 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.082572    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm" event={"ID":"03d952ac-152f-427f-85cd-9672adcc63db","Type":"ContainerStarted","Data":"24a6d0cd338b98f410d9a743793f230660ab308d4a62fc40a8a4d97587102865"}
Jul 09 17:40:36.089167 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.089146    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wznvh\""
Jul 09 17:40:36.096848 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.096825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rwpm5"
Jul 09 17:40:36.096946 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.096826    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfxts"
Jul 09 17:40:36.105290 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.105270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:36.108124 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.108101    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d4pd\""
Jul 09 17:40:36.115906 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.115843    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:36.298258 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.298229    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rwpm5"]
Jul 09 17:40:36.337816 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.337573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pfxts"]
Jul 09 17:40:36.341669 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:36.341634    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269e8c04_7a45_4450_8394_d4c231425a77.slice/crio-7cdfe8a8166776da1e333adabe55e9691bf2165c73c90101e7279c817aeb36cb WatchSource:0}: Error finding container 7cdfe8a8166776da1e333adabe55e9691bf2165c73c90101e7279c817aeb36cb: Status 404 returned error can't find the container with id 7cdfe8a8166776da1e333adabe55e9691bf2165c73c90101e7279c817aeb36cb
Jul 09 17:40:36.376831 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.376238    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bhf5x"]
Jul 09 17:40:36.381780 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:36.381740    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c84757_5b96_4311_a959_0936ed20dd67.slice/crio-6d5e826b1b78a0dbfdc0be9c303e7f962ec52d013bdd9bfc2078cdc17c610108 WatchSource:0}: Error finding container 6d5e826b1b78a0dbfdc0be9c303e7f962ec52d013bdd9bfc2078cdc17c610108: Status 404 returned error can't find the container with id 6d5e826b1b78a0dbfdc0be9c303e7f962ec52d013bdd9bfc2078cdc17c610108
Jul 09 17:40:36.402687 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:36.402603    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf9fc9d64-zx59g"]
Jul 09 17:40:36.405486 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:36.405451    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee041d1_d800_4605_92e2_4c1a91d7e487.slice/crio-0e3097075a3be5a5828bc19dd0a5136fac0dee7814bedeff8f4e532b0f3134da WatchSource:0}: Error finding container 0e3097075a3be5a5828bc19dd0a5136fac0dee7814bedeff8f4e532b0f3134da: Status 404 returned error can't find the container with id 0e3097075a3be5a5828bc19dd0a5136fac0dee7814bedeff8f4e532b0f3134da
Jul 09 17:40:37.093522 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:37.093478    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsgn4" event={"ID":"bf5c6f12-38fd-4951-bf72-18aecc136c1f","Type":"ContainerStarted","Data":"a21a4e6b72b05e1f9f6af5d65564b0efa9bf4769d09ba2f000cc8abaaaa947c3"}
Jul 09 17:40:37.095084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:37.094880    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bhf5x" event={"ID":"99c84757-5b96-4311-a959-0936ed20dd67","Type":"ContainerStarted","Data":"6d5e826b1b78a0dbfdc0be9c303e7f962ec52d013bdd9bfc2078cdc17c610108"}
Jul 09 17:40:37.096511 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:37.096486    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfxts" event={"ID":"269e8c04-7a45-4450-8394-d4c231425a77","Type":"ContainerStarted","Data":"7cdfe8a8166776da1e333adabe55e9691bf2165c73c90101e7279c817aeb36cb"}
Jul 09 17:40:37.097748 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:37.097715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rwpm5" event={"ID":"01ffe74e-2c33-429c-8818-fb8d7fa9f969","Type":"ContainerStarted","Data":"be45f3a6a5e20969ea147e5ff4e2d623a8f50c6c65f39839e8ad7b5a62b2d71e"}
Jul 09 17:40:37.099117 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:37.099095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf9fc9d64-zx59g" event={"ID":"8ee041d1-d800-4605-92e2-4c1a91d7e487","Type":"ContainerStarted","Data":"0e3097075a3be5a5828bc19dd0a5136fac0dee7814bedeff8f4e532b0f3134da"}
Jul 09 17:40:40.119291 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.119252    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsgn4" event={"ID":"bf5c6f12-38fd-4951-bf72-18aecc136c1f","Type":"ContainerStarted","Data":"dfb1c8b3d22764ae4e5643839914fbbcb4beb5388641dc32334f4be6e6e51a31"}
Jul 09 17:40:40.121607 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.121577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfxts" event={"ID":"269e8c04-7a45-4450-8394-d4c231425a77","Type":"ContainerStarted","Data":"ac6418b00bbc7b51751d18421aee3de3560034ac9aa30ee42e2d1af3bfc5e462"}
Jul 09 17:40:40.121731 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.121639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfxts" event={"ID":"269e8c04-7a45-4450-8394-d4c231425a77","Type":"ContainerStarted","Data":"2ffcf72518f54d5925b8344cf07e4a7b15bc741c6c5ef77ca02b6bc4ad21dacb"}
Jul 09 17:40:40.123714 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.123660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm" event={"ID":"03d952ac-152f-427f-85cd-9672adcc63db","Type":"ContainerStarted","Data":"59b3fdf8932a720b9e65e151adc5d68a9bfc7f50721ca0aad8e1a4ba1495dfe0"}
Jul 09 17:40:40.124014 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.123904    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"
Jul 09 17:40:40.125658 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.125609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698fc6bc8-tsvw9" event={"ID":"97cbc575-2150-49bf-a52d-49f3877d81a0","Type":"ContainerStarted","Data":"8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62"}
Jul 09 17:40:40.127948 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.127893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf9fc9d64-zx59g" event={"ID":"8ee041d1-d800-4605-92e2-4c1a91d7e487","Type":"ContainerStarted","Data":"ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9"}
Jul 09 17:40:40.130302 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.130269    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm"
Jul 09 17:40:40.144966 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.144921    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fsgn4" podStartSLOduration=1.333692334 podStartE2EDuration="11.144903725s" totalImagesPullingTime="9.811211391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:40:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.143225225 +0000 UTC m=+69.674002290" watchObservedRunningTime="2026-07-09 17:40:40.144903725 +0000 UTC m=+69.675680756"
Jul 09 17:40:40.174633 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.174591    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5698fc6bc8-tsvw9" podStartSLOduration=2.60445424 podStartE2EDuration="7.174574012s" totalImagesPullingTime="4.570119772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.172912246 +0000 UTC m=+69.703689293" watchObservedRunningTime="2026-07-09 17:40:40.174574012 +0000 UTC m=+69.705351046"
Jul 09 17:40:40.198065 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.197611    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-khplm" podStartSLOduration=1.334668098 podStartE2EDuration="5.197595394s" totalImagesPullingTime="3.862927296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.197188216 +0000 UTC m=+69.727965247" watchObservedRunningTime="2026-07-09 17:40:40.197595394 +0000 UTC m=+69.728372427"
Jul 09 17:40:40.299882 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:40.299782    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pfxts" podStartSLOduration=65.878910193 podStartE2EDuration="1m9.299761427s" totalImagesPullingTime="3.420851234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.230473388 +0000 UTC m=+69.761250421" watchObservedRunningTime="2026-07-09 17:40:40.299761427 +0000 UTC m=+69.830538458"
Jul 09 17:40:41.188930 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.188878    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bf9fc9d64-zx59g" podStartSLOduration=2.870429649 podStartE2EDuration="6.188862613s" totalImagesPullingTime="3.318432964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.299560129 +0000 UTC m=+69.830337160" watchObservedRunningTime="2026-07-09 17:40:41.188862613 +0000 UTC m=+70.719639643"
Jul 09 17:40:41.189880 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.189863    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xqhrm"]
Jul 09 17:40:41.192839 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.192821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.196542 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.196453    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 09 17:40:41.197087 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.197067    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 17:40:41.197728 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.197714    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bh67f\""
Jul 09 17:40:41.198353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.198245    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 09 17:40:41.198353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.198254    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 17:40:41.198353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.198282    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 17:40:41.213950 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.213928    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xqhrm"]
Jul 09 17:40:41.286220 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.286180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53c755e-ee57-4491-ad8b-11081e828c6c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.286349 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.286250    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgjs6\" (UniqueName: \"kubernetes.io/projected/d53c755e-ee57-4491-ad8b-11081e828c6c-kube-api-access-zgjs6\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.286349 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.286291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d53c755e-ee57-4491-ad8b-11081e828c6c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.286349 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.286320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d53c755e-ee57-4491-ad8b-11081e828c6c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.387072 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.387028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d53c755e-ee57-4491-ad8b-11081e828c6c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.387220 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.387086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d53c755e-ee57-4491-ad8b-11081e828c6c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.387220 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.387139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53c755e-ee57-4491-ad8b-11081e828c6c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.387220 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.387191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgjs6\" (UniqueName: \"kubernetes.io/projected/d53c755e-ee57-4491-ad8b-11081e828c6c-kube-api-access-zgjs6\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.387989 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.387964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53c755e-ee57-4491-ad8b-11081e828c6c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.389757 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.389732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d53c755e-ee57-4491-ad8b-11081e828c6c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.389868 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.389822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d53c755e-ee57-4491-ad8b-11081e828c6c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.399831 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.399808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgjs6\" (UniqueName: \"kubernetes.io/projected/d53c755e-ee57-4491-ad8b-11081e828c6c-kube-api-access-zgjs6\") pod \"prometheus-operator-64785b6675-xqhrm\" (UID: \"d53c755e-ee57-4491-ad8b-11081e828c6c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:41.502815 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:41.502757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm"
Jul 09 17:40:42.572914 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:42.572775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xqhrm"]
Jul 09 17:40:43.143331 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.143294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bhf5x" event={"ID":"99c84757-5b96-4311-a959-0936ed20dd67","Type":"ContainerStarted","Data":"9ba4b34702eeed1934e1b6fd06169116fef624f3714f87491c8b032325462cb5"}
Jul 09 17:40:43.143527 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.143499    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:40:43.145098 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.145071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rwpm5" event={"ID":"01ffe74e-2c33-429c-8818-fb8d7fa9f969","Type":"ContainerStarted","Data":"fdce5a1a9fac745bfa3fb02cedff988d36832af82d6506190ec0a8b0eaf53315"}
Jul 09 17:40:43.146421 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.146402    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm" event={"ID":"d53c755e-ee57-4491-ad8b-11081e828c6c","Type":"ContainerStarted","Data":"d10596f551b3e59a99fbabc8b8e02529e2fce8270bfaa65b298c02d7f425b8ef"}
Jul 09 17:40:43.161536 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.161498    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bhf5x" podStartSLOduration=66.103468958 podStartE2EDuration="1m12.161486073s" totalImagesPullingTime="6.058017115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:43.161176413 +0000 UTC m=+72.691953467" watchObservedRunningTime="2026-07-09 17:40:43.161486073 +0000 UTC m=+72.692263144"
Jul 09 17:40:43.179408 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.179365    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rwpm5" podStartSLOduration=65.032087943 podStartE2EDuration="1m11.179350318s" totalImagesPullingTime="6.147262375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:43.178482622 +0000 UTC m=+72.709259653" watchObservedRunningTime="2026-07-09 17:40:43.179350318 +0000 UTC m=+72.710127348"
Jul 09 17:40:43.730189 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.730154    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:43.730639 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.730228    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:43.735887 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:43.735865    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:44.151573 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:44.151533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm" event={"ID":"d53c755e-ee57-4491-ad8b-11081e828c6c","Type":"ContainerStarted","Data":"bef45c4faacd661a9a0ab82e043d6137be2f9adf35db23121bc7f66279123b92"}
Jul 09 17:40:44.151765 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:44.151594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm" event={"ID":"d53c755e-ee57-4491-ad8b-11081e828c6c","Type":"ContainerStarted","Data":"c4a9ff0591b6b9896a2276380ede5bff9ab47a46ac193b918f9069a0912d3f21"}
Jul 09 17:40:44.155606 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:44.155587    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:40:44.176011 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:44.175962    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-xqhrm" podStartSLOduration=1.961481519 podStartE2EDuration="3.175944186s" totalImagesPullingTime="1.214462667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:44.174221507 +0000 UTC m=+73.704998534" watchObservedRunningTime="2026-07-09 17:40:44.175944186 +0000 UTC m=+73.706721217"
Jul 09 17:40:45.084942 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:45.084914    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dzsz9"
Jul 09 17:40:46.105971 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.105937    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:46.106343 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.106221    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:46.110670 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.110649    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:46.165820 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.165798    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:40:46.242614 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.242583    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5698fc6bc8-tsvw9"]
Jul 09 17:40:46.648176 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.648147    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"]
Jul 09 17:40:46.651293 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.651276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.657452 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.657434    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 09 17:40:46.657831 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.657815    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m2wr9\""
Jul 09 17:40:46.658631 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.658610    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 09 17:40:46.661517 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.661498    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x6v6k"]
Jul 09 17:40:46.664405 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.664392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.668949 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.668928    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 17:40:46.669031 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.668967    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 17:40:46.669109 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.669024    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 17:40:46.669261 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.669246    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tbd5w\""
Jul 09 17:40:46.674246 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.674227    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-98d5p"]
Jul 09 17:40:46.677237 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.677222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.679096 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.679078    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"]
Jul 09 17:40:46.680773 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.680753    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 09 17:40:46.680918 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.680867    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 09 17:40:46.681027 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.681016    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6vqh2\""
Jul 09 17:40:46.681156 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.681093    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 09 17:40:46.706707 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.706683    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-98d5p"]
Jul 09 17:40:46.726235 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-sys\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726314 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24b99d10-5d35-406e-8805-4a9a06009a5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.726314 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz84n\" (UniqueName: \"kubernetes.io/projected/c01b5c16-251b-4550-baeb-11db543cd02a-kube-api-access-fz84n\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.726403 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzhzg\" (UniqueName: \"kubernetes.io/projected/24b99d10-5d35-406e-8805-4a9a06009a5d-kube-api-access-hzhzg\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.726403 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-tls\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726494 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726544 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24b99d10-5d35-406e-8805-4a9a06009a5d-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.726591 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726550    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcw2\" (UniqueName: \"kubernetes.io/projected/2b46e220-837a-4114-969e-84fa58778b51-kube-api-access-xqcw2\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726591 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c01b5c16-251b-4550-baeb-11db543cd02a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.726687 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b46e220-837a-4114-969e-84fa58778b51-metrics-client-ca\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726729 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-root\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726729 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-wtmp\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726819 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-accelerators-collector-config\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.726819 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.726916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.726916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-textfile\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.727008 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.726928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24b99d10-5d35-406e-8805-4a9a06009a5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.727008 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.727001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01b5c16-251b-4550-baeb-11db543cd02a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.727259 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.727130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.828368 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-tls\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24b99d10-5d35-406e-8805-4a9a06009a5d-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.828475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcw2\" (UniqueName: \"kubernetes.io/projected/2b46e220-837a-4114-969e-84fa58778b51-kube-api-access-xqcw2\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c01b5c16-251b-4550-baeb-11db543cd02a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.828475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b46e220-837a-4114-969e-84fa58778b51-metrics-client-ca\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-root\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-wtmp\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-accelerators-collector-config\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828593    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-root\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-textfile\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24b99d10-5d35-406e-8805-4a9a06009a5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.828702 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01b5c16-251b-4550-baeb-11db543cd02a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-sys\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24b99d10-5d35-406e-8805-4a9a06009a5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz84n\" (UniqueName: \"kubernetes.io/projected/c01b5c16-251b-4550-baeb-11db543cd02a-kube-api-access-fz84n\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzhzg\" (UniqueName: \"kubernetes.io/projected/24b99d10-5d35-406e-8805-4a9a06009a5d-kube-api-access-hzhzg\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828866    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c01b5c16-251b-4550-baeb-11db543cd02a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.828995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-wtmp\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.829158    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b46e220-837a-4114-969e-84fa58778b51-metrics-client-ca\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.829209 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.829190    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24b99d10-5d35-406e-8805-4a9a06009a5d-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.829655 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.829462    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-textfile\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.829655 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.829515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-accelerators-collector-config\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.829655 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.829580    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b46e220-837a-4114-969e-84fa58778b51-sys\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.829805 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.829772    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01b5c16-251b-4550-baeb-11db543cd02a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.830323 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.830300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.831485 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.831460    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.831633 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.831615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b46e220-837a-4114-969e-84fa58778b51-node-exporter-tls\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.831690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.831628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.831886 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.831865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01b5c16-251b-4550-baeb-11db543cd02a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.831941 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.831926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/24b99d10-5d35-406e-8805-4a9a06009a5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.832012 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.831995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24b99d10-5d35-406e-8805-4a9a06009a5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.840633 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.840611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz84n\" (UniqueName: \"kubernetes.io/projected/c01b5c16-251b-4550-baeb-11db543cd02a-kube-api-access-fz84n\") pod \"kube-state-metrics-785778ff74-98d5p\" (UID: \"c01b5c16-251b-4550-baeb-11db543cd02a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:46.840973 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.840948    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzhzg\" (UniqueName: \"kubernetes.io/projected/24b99d10-5d35-406e-8805-4a9a06009a5d-kube-api-access-hzhzg\") pod \"openshift-state-metrics-84bd68b896-v6nj5\" (UID: \"24b99d10-5d35-406e-8805-4a9a06009a5d\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.841271 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.841252    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcw2\" (UniqueName: \"kubernetes.io/projected/2b46e220-837a-4114-969e-84fa58778b51-kube-api-access-xqcw2\") pod \"node-exporter-x6v6k\" (UID: \"2b46e220-837a-4114-969e-84fa58778b51\") " pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.959555 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.959512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"
Jul 09 17:40:46.971945 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.971921    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x6v6k"
Jul 09 17:40:46.982689 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:46.982668    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b46e220_837a_4114_969e_84fa58778b51.slice/crio-d97b524771c347b28ac3660ab272ee2d676acb110fb4f51a951787ec32de71b1 WatchSource:0}: Error finding container d97b524771c347b28ac3660ab272ee2d676acb110fb4f51a951787ec32de71b1: Status 404 returned error can't find the container with id d97b524771c347b28ac3660ab272ee2d676acb110fb4f51a951787ec32de71b1
Jul 09 17:40:46.986003 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:46.985986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p"
Jul 09 17:40:47.104236 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.104209    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5"]
Jul 09 17:40:47.107570 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:47.107542    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b99d10_5d35_406e_8805_4a9a06009a5d.slice/crio-c7d0358174369c7f11bc5d9b5153d86fed63900e1534b2ace85804c6910f1c45 WatchSource:0}: Error finding container c7d0358174369c7f11bc5d9b5153d86fed63900e1534b2ace85804c6910f1c45: Status 404 returned error can't find the container with id c7d0358174369c7f11bc5d9b5153d86fed63900e1534b2ace85804c6910f1c45
Jul 09 17:40:47.135627 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.135601    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-98d5p"]
Jul 09 17:40:47.138096 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:47.138072    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01b5c16_251b_4550_baeb_11db543cd02a.slice/crio-a27f4556cc093f3b9bf8279c28cb4779513c0ffdeefd69950383d35b26edd320 WatchSource:0}: Error finding container a27f4556cc093f3b9bf8279c28cb4779513c0ffdeefd69950383d35b26edd320: Status 404 returned error can't find the container with id a27f4556cc093f3b9bf8279c28cb4779513c0ffdeefd69950383d35b26edd320
Jul 09 17:40:47.166901 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.166871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5" event={"ID":"24b99d10-5d35-406e-8805-4a9a06009a5d","Type":"ContainerStarted","Data":"c7d0358174369c7f11bc5d9b5153d86fed63900e1534b2ace85804c6910f1c45"}
Jul 09 17:40:47.168014 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.167992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p" event={"ID":"c01b5c16-251b-4550-baeb-11db543cd02a","Type":"ContainerStarted","Data":"a27f4556cc093f3b9bf8279c28cb4779513c0ffdeefd69950383d35b26edd320"}
Jul 09 17:40:47.169318 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.169295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6v6k" event={"ID":"2b46e220-837a-4114-969e-84fa58778b51","Type":"ContainerStarted","Data":"d97b524771c347b28ac3660ab272ee2d676acb110fb4f51a951787ec32de71b1"}
Jul 09 17:40:47.851061 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.849393    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 17:40:47.855225 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.855194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.860400 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.860376    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 17:40:47.860539 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.860461    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 17:40:47.860539 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.860469    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 17:40:47.861127 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.861108    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 17:40:47.861220 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.861012    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 17:40:47.861286 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.861031    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rb9gm\""
Jul 09 17:40:47.861286 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.861107    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 17:40:47.861404 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.861327    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 17:40:47.861404 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.860992    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 17:40:47.861525 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.861396    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 17:40:47.900167 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.900070    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 17:40:47.939098 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.938968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c683809c-af44-4756-9c3c-63e979924bda-config-out\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939098 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-config-volume\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939098 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939333 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939333 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c683809c-af44-4756-9c3c-63e979924bda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939333 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-web-config\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c683809c-af44-4756-9c3c-63e979924bda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c683809c-af44-4756-9c3c-63e979924bda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939475 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939714 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbtw\" (UniqueName: \"kubernetes.io/projected/c683809c-af44-4756-9c3c-63e979924bda-kube-api-access-qjbtw\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:47.939714 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:47.939542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c683809c-af44-4756-9c3c-63e979924bda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.040320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.040292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-config-volume\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.040468 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.040343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.040468 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:48.040450    2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 09 17:40:48.040575 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:48.040511    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-main-tls podName:c683809c-af44-4756-9c3c-63e979924bda nodeName:}" failed. No retries permitted until 2026-07-09 17:40:48.540489503 +0000 UTC m=+78.071266513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c683809c-af44-4756-9c3c-63e979924bda") : secret "alertmanager-main-tls" not found
Jul 09 17:40:48.040750 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.040716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.040855 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.040770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c683809c-af44-4756-9c3c-63e979924bda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.040855 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.040802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-web-config\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.040855 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.040848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.041000 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.040881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c683809c-af44-4756-9c3c-63e979924bda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.041337 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041284    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c683809c-af44-4756-9c3c-63e979924bda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c683809c-af44-4756-9c3c-63e979924bda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c683809c-af44-4756-9c3c-63e979924bda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbtw\" (UniqueName: \"kubernetes.io/projected/c683809c-af44-4756-9c3c-63e979924bda-kube-api-access-qjbtw\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c683809c-af44-4756-9c3c-63e979924bda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.041988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c683809c-af44-4756-9c3c-63e979924bda-config-out\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.042688 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.042297    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c683809c-af44-4756-9c3c-63e979924bda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.045360 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.045338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c683809c-af44-4756-9c3c-63e979924bda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.045464 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.045399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.046084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.045881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c683809c-af44-4756-9c3c-63e979924bda-config-out\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.046084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.045898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.046084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.046027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.046326 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.046303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-config-volume\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.047329 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.047291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-web-config\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.047928 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.047752    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.063232 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.063165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbtw\" (UniqueName: \"kubernetes.io/projected/c683809c-af44-4756-9c3c-63e979924bda-kube-api-access-qjbtw\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.175828 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.175747    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5" event={"ID":"24b99d10-5d35-406e-8805-4a9a06009a5d","Type":"ContainerStarted","Data":"47257a80a690b41a02810e4279aaac7caed97065200ea98887f42f6d895fb98e"}
Jul 09 17:40:48.175828 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.175822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5" event={"ID":"24b99d10-5d35-406e-8805-4a9a06009a5d","Type":"ContainerStarted","Data":"6c448b1660678d75c4ba3efc0aed36ac496d153ebed689a9a9628728be084716"}
Jul 09 17:40:48.177588 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.177562    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6v6k" event={"ID":"2b46e220-837a-4114-969e-84fa58778b51","Type":"ContainerStarted","Data":"9cdbf4a6a18c7f3d16bff98afe033554af34965206162d7b8035d0a553b4ed82"}
Jul 09 17:40:48.547391 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.547320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.549544 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.549516    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c683809c-af44-4756-9c3c-63e979924bda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c683809c-af44-4756-9c3c-63e979924bda\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.767470 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.767437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 17:40:48.928137 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:48.928024    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 17:40:48.929991 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:48.929959    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc683809c_af44_4756_9c3c_63e979924bda.slice/crio-6fe5caced66c0b8e3c16c90bf97c408b1f8891a3933cf6b35bf0405311c4435b WatchSource:0}: Error finding container 6fe5caced66c0b8e3c16c90bf97c408b1f8891a3933cf6b35bf0405311c4435b: Status 404 returned error can't find the container with id 6fe5caced66c0b8e3c16c90bf97c408b1f8891a3933cf6b35bf0405311c4435b
Jul 09 17:40:49.183257 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.183170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5" event={"ID":"24b99d10-5d35-406e-8805-4a9a06009a5d","Type":"ContainerStarted","Data":"e0cb447e9f4bcbd700ea726fe634b8ab66363e15de129a72e2f0eb1feb6e2f80"}
Jul 09 17:40:49.185221 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.185197    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p" event={"ID":"c01b5c16-251b-4550-baeb-11db543cd02a","Type":"ContainerStarted","Data":"1184ab2a4bac39ac5e4c7f1189d33352caa8c5c73333554e250aa7aaf3933600"}
Jul 09 17:40:49.185346 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.185246    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p" event={"ID":"c01b5c16-251b-4550-baeb-11db543cd02a","Type":"ContainerStarted","Data":"0320694358f0088fd084d424dee6eb409828146a497eaf90e9f6ebd93dcb25de"}
Jul 09 17:40:49.185346 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.185275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p" event={"ID":"c01b5c16-251b-4550-baeb-11db543cd02a","Type":"ContainerStarted","Data":"9603c22cad58f4edb8eb254c400305c65c090709ce6a4b3458e697cc3a1649f5"}
Jul 09 17:40:49.186342 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.186315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerStarted","Data":"6fe5caced66c0b8e3c16c90bf97c408b1f8891a3933cf6b35bf0405311c4435b"}
Jul 09 17:40:49.187690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.187669    2576 generic.go:356] "Generic (PLEG): container finished" podID="2b46e220-837a-4114-969e-84fa58778b51" containerID="9cdbf4a6a18c7f3d16bff98afe033554af34965206162d7b8035d0a553b4ed82" exitCode=0
Jul 09 17:40:49.187769 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.187712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6v6k" event={"ID":"2b46e220-837a-4114-969e-84fa58778b51","Type":"ContainerDied","Data":"9cdbf4a6a18c7f3d16bff98afe033554af34965206162d7b8035d0a553b4ed82"}
Jul 09 17:40:49.208933 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.208868    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-v6nj5" podStartSLOduration=1.8878265459999999 podStartE2EDuration="3.208850154s" totalImagesPullingTime="1.321023608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:49.208563134 +0000 UTC m=+78.739340166" watchObservedRunningTime="2026-07-09 17:40:49.208850154 +0000 UTC m=+78.739627185"
Jul 09 17:40:49.274270 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.274225    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-98d5p" podStartSLOduration=1.8519065220000002 podStartE2EDuration="3.274196208s" totalImagesPullingTime="1.422289686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:49.272793216 +0000 UTC m=+78.803570247" watchObservedRunningTime="2026-07-09 17:40:49.274196208 +0000 UTC m=+78.804973238"
Jul 09 17:40:49.802472 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.802442    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6779965f77-zwksh"]
Jul 09 17:40:49.806170 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.806146    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.811812 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.811787    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 17:40:49.811970 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.811951    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 17:40:49.812030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.811975    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 17:40:49.812130 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.812116    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 17:40:49.812348 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.812332    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kgktc\""
Jul 09 17:40:49.813192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.813165    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-47mafo8fl3c8h\""
Jul 09 17:40:49.813924 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.813903    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 17:40:49.824778 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.824754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6779965f77-zwksh"]
Jul 09 17:40:49.861116 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-tls\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.861215 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.861215 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.861316 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-grpc-tls\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.861316 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.861316 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rcd\" (UniqueName: \"kubernetes.io/projected/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-kube-api-access-c5rcd\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.861449 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-metrics-client-ca\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.861449 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.861370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962472 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rcd\" (UniqueName: \"kubernetes.io/projected/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-kube-api-access-c5rcd\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962472 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-metrics-client-ca\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962630 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962630 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-tls\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962630 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962790 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962790 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-grpc-tls\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.962790 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.962764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.963291 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.963238    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-metrics-client-ca\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.965008 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.964901    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.965354 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.965309    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.965474 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.965449    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.965552 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.965499    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-grpc-tls\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.965552 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.965520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.965652 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.965607    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-secret-thanos-querier-tls\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:49.981417 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:49.981399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rcd\" (UniqueName: \"kubernetes.io/projected/8f5e947c-e2d1-4028-8fb5-b695d7716e0e-kube-api-access-c5rcd\") pod \"thanos-querier-6779965f77-zwksh\" (UID: \"8f5e947c-e2d1-4028-8fb5-b695d7716e0e\") " pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:50.115167 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:50.115144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:50.194797 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:50.194764    2576 generic.go:356] "Generic (PLEG): container finished" podID="c683809c-af44-4756-9c3c-63e979924bda" containerID="8d6df27b4a5def11cf63c76ee3e56d074e93ed78cddcc00ef6ce9440b33349d8" exitCode=0
Jul 09 17:40:50.195228 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:50.194998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerDied","Data":"8d6df27b4a5def11cf63c76ee3e56d074e93ed78cddcc00ef6ce9440b33349d8"}
Jul 09 17:40:50.199593 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:50.199564    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6v6k" event={"ID":"2b46e220-837a-4114-969e-84fa58778b51","Type":"ContainerStarted","Data":"830cc2450a64b27135cfa64c77bd6ec6f9982a556935318bf7fdf853d9cbb694"}
Jul 09 17:40:50.199678 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:50.199629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x6v6k" event={"ID":"2b46e220-837a-4114-969e-84fa58778b51","Type":"ContainerStarted","Data":"3fd96b73fd5a6ab10f634462216dd30db2cb4800656c2a86d7c12255d21b0e17"}
Jul 09 17:40:50.251352 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:50.251301    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x6v6k" podStartSLOduration=3.287733902 podStartE2EDuration="4.25128248s" totalImagesPullingTime="885.90363ms" totalInitContainerRuntime="77.644948ms" isStatefulPod=true podCreationTimestamp="2026-07-09 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:50.250477663 +0000 UTC m=+79.781254694" watchObservedRunningTime="2026-07-09 17:40:50.25128248 +0000 UTC m=+79.782059512"
Jul 09 17:40:50.273357 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:50.273330    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6779965f77-zwksh"]
Jul 09 17:40:50.275539 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:50.275517    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5e947c_e2d1_4028_8fb5_b695d7716e0e.slice/crio-000bc113e7349fc6fc91f8bc418b4973a69aef3ad9319f1522667260fb347534 WatchSource:0}: Error finding container 000bc113e7349fc6fc91f8bc418b4973a69aef3ad9319f1522667260fb347534: Status 404 returned error can't find the container with id 000bc113e7349fc6fc91f8bc418b4973a69aef3ad9319f1522667260fb347534
Jul 09 17:40:51.060301 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.060271    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dbc88fcb8-4rjtv"
Jul 09 17:40:51.189430 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.189401    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5975c47848-c26vd"]
Jul 09 17:40:51.192906 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.192885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.196785 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.196517    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 17:40:51.196785 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.196526    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 17:40:51.196785 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.196609    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 17:40:51.196785 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.196641    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dsxzs\""
Jul 09 17:40:51.196785 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.196752    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 17:40:51.198267 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.198092    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f9k57psebnsjc\""
Jul 09 17:40:51.207878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.207851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" event={"ID":"8f5e947c-e2d1-4028-8fb5-b695d7716e0e","Type":"ContainerStarted","Data":"000bc113e7349fc6fc91f8bc418b4973a69aef3ad9319f1522667260fb347534"}
Jul 09 17:40:51.213491 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.213466    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5975c47848-c26vd"]
Jul 09 17:40:51.275569 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.275538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8q9j\" (UniqueName: \"kubernetes.io/projected/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-kube-api-access-h8q9j\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.275690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.275613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-secret-metrics-server-tls\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.275690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.275653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-secret-metrics-server-client-certs\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.275690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.275680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-metrics-server-audit-profiles\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.275826 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.275703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-client-ca-bundle\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.275826 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.275770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-audit-log\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.275826 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.275807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.376678 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.376648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8q9j\" (UniqueName: \"kubernetes.io/projected/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-kube-api-access-h8q9j\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.376839 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.376710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-secret-metrics-server-tls\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.376839 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.376748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-secret-metrics-server-client-certs\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.376839 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.376780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-metrics-server-audit-profiles\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.376839 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.376800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-client-ca-bundle\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.376839 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.376830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-audit-log\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.377144 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.376861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.377686 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.377655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.377936 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.377917    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-audit-log\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.378428 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.378362    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-metrics-server-audit-profiles\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.380501 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.380444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-client-ca-bundle\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.381148 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.381124    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-secret-metrics-server-tls\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.381253 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.381127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-secret-metrics-server-client-certs\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.401564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.401536    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"]
Jul 09 17:40:51.404142 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.404096    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8q9j\" (UniqueName: \"kubernetes.io/projected/b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8-kube-api-access-h8q9j\") pod \"metrics-server-5975c47848-c26vd\" (UID: \"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8\") " pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.406666 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.406646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:51.409858 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.409838    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dfj6x\""
Jul 09 17:40:51.409965 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.409949    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 09 17:40:51.418609 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.418557    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"]
Jul 09 17:40:51.478286 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.478259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5fea503-e58d-4f3c-8e54-9e5441ab0202-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x7rcz\" (UID: \"d5fea503-e58d-4f3c-8e54-9e5441ab0202\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:51.478397 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.478295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hgf\" (UniqueName: \"kubernetes.io/projected/d5fea503-e58d-4f3c-8e54-9e5441ab0202-kube-api-access-95hgf\") pod \"monitoring-plugin-5f68df7f58-x7rcz\" (UID: \"d5fea503-e58d-4f3c-8e54-9e5441ab0202\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:51.506184 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.506157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:40:51.579358 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.579333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hgf\" (UniqueName: \"kubernetes.io/projected/d5fea503-e58d-4f3c-8e54-9e5441ab0202-kube-api-access-95hgf\") pod \"monitoring-plugin-5f68df7f58-x7rcz\" (UID: \"d5fea503-e58d-4f3c-8e54-9e5441ab0202\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:51.579499 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.579462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5fea503-e58d-4f3c-8e54-9e5441ab0202-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x7rcz\" (UID: \"d5fea503-e58d-4f3c-8e54-9e5441ab0202\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:51.579564 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:51.579555    2576 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 09 17:40:51.579641 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:40:51.579628    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fea503-e58d-4f3c-8e54-9e5441ab0202-monitoring-plugin-cert podName:d5fea503-e58d-4f3c-8e54-9e5441ab0202 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:52.079609494 +0000 UTC m=+81.610386520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d5fea503-e58d-4f3c-8e54-9e5441ab0202-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-x7rcz" (UID: "d5fea503-e58d-4f3c-8e54-9e5441ab0202") : secret "monitoring-plugin-cert" not found
Jul 09 17:40:51.589090 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.588989    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hgf\" (UniqueName: \"kubernetes.io/projected/d5fea503-e58d-4f3c-8e54-9e5441ab0202-kube-api-access-95hgf\") pod \"monitoring-plugin-5f68df7f58-x7rcz\" (UID: \"d5fea503-e58d-4f3c-8e54-9e5441ab0202\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:51.723069 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:51.723020    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5975c47848-c26vd"]
Jul 09 17:40:51.727913 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:51.727883    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ac9b34_3e1e_40e2_bda2_0d3a0ea6c2c8.slice/crio-2d3aa22177fd75107b4ae64e58c44b49af61e7d02ccd90f1559c1ab6a9384571 WatchSource:0}: Error finding container 2d3aa22177fd75107b4ae64e58c44b49af61e7d02ccd90f1559c1ab6a9384571: Status 404 returned error can't find the container with id 2d3aa22177fd75107b4ae64e58c44b49af61e7d02ccd90f1559c1ab6a9384571
Jul 09 17:40:52.083758 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:52.083715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5fea503-e58d-4f3c-8e54-9e5441ab0202-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x7rcz\" (UID: \"d5fea503-e58d-4f3c-8e54-9e5441ab0202\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:52.086286 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:52.086266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5fea503-e58d-4f3c-8e54-9e5441ab0202-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-x7rcz\" (UID: \"d5fea503-e58d-4f3c-8e54-9e5441ab0202\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:52.214696 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:52.214660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerStarted","Data":"f181f73bc1afa3bcc9c9ebf918f112c8882b57df8c0a80bd6205687047f746a4"}
Jul 09 17:40:52.215131 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:52.214775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerStarted","Data":"305f9302575e9b77d5fead512d793e89472449bf9e173a1f4c33ca98da569834"}
Jul 09 17:40:52.215969 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:52.215944    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5975c47848-c26vd" event={"ID":"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8","Type":"ContainerStarted","Data":"2d3aa22177fd75107b4ae64e58c44b49af61e7d02ccd90f1559c1ab6a9384571"}
Jul 09 17:40:52.324165 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:52.324140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:52.496868 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:52.496796    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"]
Jul 09 17:40:52.503483 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:52.503446    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fea503_e58d_4f3c_8e54_9e5441ab0202.slice/crio-b624bc463b452fc06721486a234249b977fb5b5bb09d1df047fdb308c83aec98 WatchSource:0}: Error finding container b624bc463b452fc06721486a234249b977fb5b5bb09d1df047fdb308c83aec98: Status 404 returned error can't find the container with id b624bc463b452fc06721486a234249b977fb5b5bb09d1df047fdb308c83aec98
Jul 09 17:40:53.224457 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.224412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" event={"ID":"8f5e947c-e2d1-4028-8fb5-b695d7716e0e","Type":"ContainerStarted","Data":"679d6fb5b133e23a6dee91376dd74327b59ce0c3385eefc90d6fad8ae405576f"}
Jul 09 17:40:53.224949 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.224494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" event={"ID":"8f5e947c-e2d1-4028-8fb5-b695d7716e0e","Type":"ContainerStarted","Data":"ca016a0218e65a291bec7af07eebe065c1b19a3d30b90ebba0bb75897994fc4f"}
Jul 09 17:40:53.224949 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.224530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" event={"ID":"8f5e947c-e2d1-4028-8fb5-b695d7716e0e","Type":"ContainerStarted","Data":"a6f82578148a39b785a6a6a828ff27e385cba7b1b963f1313ffee584548c2c49"}
Jul 09 17:40:53.228292 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.228268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerStarted","Data":"d09b2f2581e5ba017c7ccb47be41aa4ee9a30510a2bcb9f8ab3bd686e28eb26a"}
Jul 09 17:40:53.228425 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.228334    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerStarted","Data":"82976e9e66551a915f0be7f0fff4da7a530d013d47b3b6056006b90632ba54fd"}
Jul 09 17:40:53.228425 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.228379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerStarted","Data":"b2f3d4bec07d36e1c0e64d68a4ad9230ad92cfb8dccb7f2d8333cad5c56c9088"}
Jul 09 17:40:53.229628 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.229606    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz" event={"ID":"d5fea503-e58d-4f3c-8e54-9e5441ab0202","Type":"ContainerStarted","Data":"b624bc463b452fc06721486a234249b977fb5b5bb09d1df047fdb308c83aec98"}
Jul 09 17:40:53.306068 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.304737    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 17:40:53.310833 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.309495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.313265 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.313243    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 17:40:53.313363 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.313243    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 17:40:53.313420 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.313400    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 17:40:53.315508 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.315487    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 17:40:53.315615 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.315545    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 17:40:53.315615 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.315566    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 17:40:53.319889 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.319193    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 17:40:53.319889 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.319407    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h42xt\""
Jul 09 17:40:53.319889 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.319717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 17:40:53.319889 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.319761    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 17:40:53.319889 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.319857    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 17:40:53.319889 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.319874    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 17:40:53.320205 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.319929    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d096oqp7qhme\""
Jul 09 17:40:53.320837 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.320819    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 17:40:53.321000 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.320982    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 17:40:53.322850 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.322832    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 17:40:53.351953 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.351934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 17:40:53.397606 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.397756 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-config\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.397756 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32ea468c-e785-4180-bdb1-a473f520f40c-config-out\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.397756 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.397756 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.397756 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32ea468c-e785-4180-bdb1-a473f520f40c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398007 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398007 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.397817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398136 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-web-config\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398136 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398284 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398284 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zpq\" (UniqueName: \"kubernetes.io/projected/32ea468c-e785-4180-bdb1-a473f520f40c-kube-api-access-59zpq\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398284 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398284 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.398693 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.398509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499460 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-web-config\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499460 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499660 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499660 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zpq\" (UniqueName: \"kubernetes.io/projected/32ea468c-e785-4180-bdb1-a473f520f40c-kube-api-access-59zpq\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499660 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499846 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499921 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.499988 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.499971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500069 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500130 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500177 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500224 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500224 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-config\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32ea468c-e785-4180-bdb1-a473f520f40c-config-out\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500450 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500450 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32ea468c-e785-4180-bdb1-a473f520f40c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500450 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.500450 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.500437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.504353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.501285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.504353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.502126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.504353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.503274    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.504353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.503874    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.504353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.504193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.505328 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.505307    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.505697 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.505590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.511555 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.511528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.512157 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.512133    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.512157 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.512147    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32ea468c-e785-4180-bdb1-a473f520f40c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.512373 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.512351    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-web-config\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.512793 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.512770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.512920 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.512898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.513742 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.513712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zpq\" (UniqueName: \"kubernetes.io/projected/32ea468c-e785-4180-bdb1-a473f520f40c-kube-api-access-59zpq\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.515775 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.515750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32ea468c-e785-4180-bdb1-a473f520f40c-config-out\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.516031 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.516007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.518165 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.518139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-config\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.518279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.518189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32ea468c-e785-4180-bdb1-a473f520f40c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.519697 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.519635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.520580 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.520559    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32ea468c-e785-4180-bdb1-a473f520f40c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32ea468c-e785-4180-bdb1-a473f520f40c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.624001 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.623971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:40:53.814919 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:53.814827    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 17:40:53.820167 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:40:53.820132    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ea468c_e785_4180_bdb1_a473f520f40c.slice/crio-34a5861a73c9d3b073954d9cce076f50bd8d6d40455def79a67ea3baf4445900 WatchSource:0}: Error finding container 34a5861a73c9d3b073954d9cce076f50bd8d6d40455def79a67ea3baf4445900: Status 404 returned error can't find the container with id 34a5861a73c9d3b073954d9cce076f50bd8d6d40455def79a67ea3baf4445900
Jul 09 17:40:54.235379 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.235349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"94bbb18b1571511d18f6b744ac4d7e401e71cb9a103bd84ba7d917f37582ca17"}
Jul 09 17:40:54.235800 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.235429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"34a5861a73c9d3b073954d9cce076f50bd8d6d40455def79a67ea3baf4445900"}
Jul 09 17:40:54.236982 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.236952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5975c47848-c26vd" event={"ID":"b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8","Type":"ContainerStarted","Data":"272f190cfb3939435fc7422e7ae053f0544a21dbdf38b7c4de847986f762aa9b"}
Jul 09 17:40:54.239796 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.239775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" event={"ID":"8f5e947c-e2d1-4028-8fb5-b695d7716e0e","Type":"ContainerStarted","Data":"85d91efea6669cdb6ab163066a6ad67fd7431945bbe269a1eb190540188a0577"}
Jul 09 17:40:54.239896 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.239836    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" event={"ID":"8f5e947c-e2d1-4028-8fb5-b695d7716e0e","Type":"ContainerStarted","Data":"1fc237f1e8243bab4ffd7cee8d1ca84459fbe3f1dad88c42d61cf4304ba526b4"}
Jul 09 17:40:54.239896 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.239878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" event={"ID":"8f5e947c-e2d1-4028-8fb5-b695d7716e0e","Type":"ContainerStarted","Data":"a498adee0c47d42a505f8494694e898f751bcdbe2ea81cf5a660ac20f062ee35"}
Jul 09 17:40:54.240024 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.239918    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:40:54.243392 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.243359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c683809c-af44-4756-9c3c-63e979924bda","Type":"ContainerStarted","Data":"fd6442d3298a4387c9bbe4f7c207f5a9a97660536331effe8c3f4aae4fec8101"}
Jul 09 17:40:54.295009 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.294972    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh" podStartSLOduration=2.205495965 podStartE2EDuration="5.294957106s" totalImagesPullingTime="3.089461141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:54.293009604 +0000 UTC m=+83.823786635" watchObservedRunningTime="2026-07-09 17:40:54.294957106 +0000 UTC m=+83.825734136"
Jul 09 17:40:54.315940 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.315895    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5975c47848-c26vd" podStartSLOduration=1.69957688 podStartE2EDuration="3.315875849s" totalImagesPullingTime="1.616298969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:54.313425847 +0000 UTC m=+83.844202871" watchObservedRunningTime="2026-07-09 17:40:54.315875849 +0000 UTC m=+83.846652880"
Jul 09 17:40:54.352074 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:54.352009    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.92073813 podStartE2EDuration="7.351991663s" totalImagesPullingTime="3.415645084s" totalInitContainerRuntime="15.608449ms" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:54.351433959 +0000 UTC m=+83.882210990" watchObservedRunningTime="2026-07-09 17:40:54.351991663 +0000 UTC m=+83.882768694"
Jul 09 17:40:55.249297 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:55.249259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz" event={"ID":"d5fea503-e58d-4f3c-8e54-9e5441ab0202","Type":"ContainerStarted","Data":"2ee269cf21d29309d844fcd35384ea6d8e308ebc1bf5c6f9f0cabeee6cea1796"}
Jul 09 17:40:55.249750 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:55.249442    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:55.250901 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:55.250875    2576 generic.go:356] "Generic (PLEG): container finished" podID="32ea468c-e785-4180-bdb1-a473f520f40c" containerID="94bbb18b1571511d18f6b744ac4d7e401e71cb9a103bd84ba7d917f37582ca17" exitCode=0
Jul 09 17:40:55.251029 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:55.250982    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerDied","Data":"94bbb18b1571511d18f6b744ac4d7e401e71cb9a103bd84ba7d917f37582ca17"}
Jul 09 17:40:55.254693 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:55.254673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz"
Jul 09 17:40:55.267618 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:55.267576    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-x7rcz" podStartSLOduration=2.481626005 podStartE2EDuration="4.267561507s" totalImagesPullingTime="1.785935502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:55.266576724 +0000 UTC m=+84.797353757" watchObservedRunningTime="2026-07-09 17:40:55.267561507 +0000 UTC m=+84.798338536"
Jul 09 17:40:58.272487 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:58.272451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"9745f6bd320659bb0e6836a36fa49db3eed2a88742403525df7ea76816504e08"}
Jul 09 17:40:59.280279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:59.280248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"c3ba3b56fa79784db350602d6a7e8f9b75dd710dabcf85795d62626e11d23332"}
Jul 09 17:40:59.280655 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:59.280320    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"e01f3a2f514b3e95af7e3113db2895191f16109e1500f1d373513bccfa791c62"}
Jul 09 17:40:59.280655 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:59.280362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"7c08d3b51cbf9f6fcde40444e2ae9b4f1e0c2aa7e1a41fb6236506be03278ead"}
Jul 09 17:40:59.280655 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:59.280404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"14e5cd06d44a4df1f09263d712ac64178d16c4a5912ca996ac4f95a941ad0416"}
Jul 09 17:40:59.280655 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:59.280444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32ea468c-e785-4180-bdb1-a473f520f40c","Type":"ContainerStarted","Data":"bf8763dd07c84d815153af1474c26b7e699860f4eb070d7b7b5a5535b420b8a7"}
Jul 09 17:40:59.319749 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:40:59.319692    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.405481472 podStartE2EDuration="6.319677685s" totalImagesPullingTime="2.889444875s" totalInitContainerRuntime="24.751338ms" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:59.317641423 +0000 UTC m=+88.848418450" watchObservedRunningTime="2026-07-09 17:40:59.319677685 +0000 UTC m=+88.850454715"
Jul 09 17:41:00.256673 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:00.256647    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6779965f77-zwksh"
Jul 09 17:41:03.511469 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:03.511436    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf9fc9d64-zx59g"]
Jul 09 17:41:03.624320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:03.624292    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:41:11.269565 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.269516    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5698fc6bc8-tsvw9" podUID="97cbc575-2150-49bf-a52d-49f3877d81a0" containerName="console" containerID="cri-o://8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62" gracePeriod=15
Jul 09 17:41:11.506022 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.506001    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:41:11.506266 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.506240    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:41:11.506325 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.506298    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:41:11.652240 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652214    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-oauth-config\") pod \"97cbc575-2150-49bf-a52d-49f3877d81a0\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") "
Jul 09 17:41:11.652370 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652253    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-oauth-serving-cert\") pod \"97cbc575-2150-49bf-a52d-49f3877d81a0\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") "
Jul 09 17:41:11.652370 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652303    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-console-config\") pod \"97cbc575-2150-49bf-a52d-49f3877d81a0\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") "
Jul 09 17:41:11.652370 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652361    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-serving-cert\") pod \"97cbc575-2150-49bf-a52d-49f3877d81a0\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") "
Jul 09 17:41:11.652526 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqbhm\" (UniqueName: \"kubernetes.io/projected/97cbc575-2150-49bf-a52d-49f3877d81a0-kube-api-access-wqbhm\") pod \"97cbc575-2150-49bf-a52d-49f3877d81a0\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") "
Jul 09 17:41:11.652526 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652459    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-service-ca\") pod \"97cbc575-2150-49bf-a52d-49f3877d81a0\" (UID: \"97cbc575-2150-49bf-a52d-49f3877d81a0\") "
Jul 09 17:41:11.652816 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652746    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-console-config" (OuterVolumeSpecName: "console-config") pod "97cbc575-2150-49bf-a52d-49f3877d81a0" (UID: "97cbc575-2150-49bf-a52d-49f3877d81a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:11.652816 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652754    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97cbc575-2150-49bf-a52d-49f3877d81a0" (UID: "97cbc575-2150-49bf-a52d-49f3877d81a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:11.652987 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.652840    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "97cbc575-2150-49bf-a52d-49f3877d81a0" (UID: "97cbc575-2150-49bf-a52d-49f3877d81a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:11.653132 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.653113    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-service-ca\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:11.653210 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.653137    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-oauth-serving-cert\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:11.653210 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.653153    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97cbc575-2150-49bf-a52d-49f3877d81a0-console-config\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:11.654711 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.654686    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97cbc575-2150-49bf-a52d-49f3877d81a0" (UID: "97cbc575-2150-49bf-a52d-49f3877d81a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:41:11.654711 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.654694    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cbc575-2150-49bf-a52d-49f3877d81a0-kube-api-access-wqbhm" (OuterVolumeSpecName: "kube-api-access-wqbhm") pod "97cbc575-2150-49bf-a52d-49f3877d81a0" (UID: "97cbc575-2150-49bf-a52d-49f3877d81a0"). InnerVolumeSpecName "kube-api-access-wqbhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:41:11.654856 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.654773    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97cbc575-2150-49bf-a52d-49f3877d81a0" (UID: "97cbc575-2150-49bf-a52d-49f3877d81a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:41:11.758580 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.754956    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-oauth-config\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:11.758580 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.754985    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97cbc575-2150-49bf-a52d-49f3877d81a0-console-serving-cert\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:11.758580 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:11.755006    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqbhm\" (UniqueName: \"kubernetes.io/projected/97cbc575-2150-49bf-a52d-49f3877d81a0-kube-api-access-wqbhm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:12.339603 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.339571    2576 generic.go:356] "Generic (PLEG): container finished" podID="97cbc575-2150-49bf-a52d-49f3877d81a0" containerID="8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62" exitCode=0
Jul 09 17:41:12.339961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.339633    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5698fc6bc8-tsvw9"
Jul 09 17:41:12.339961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.339663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698fc6bc8-tsvw9" event={"ID":"97cbc575-2150-49bf-a52d-49f3877d81a0","Type":"ContainerDied","Data":"8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62"}
Jul 09 17:41:12.339961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.339745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5698fc6bc8-tsvw9" event={"ID":"97cbc575-2150-49bf-a52d-49f3877d81a0","Type":"ContainerDied","Data":"6de7e3b9ac16181b4c0b9a9de0e5d53cfb70f7c1a7f2d9dbcec9cbe84a2dfcce"}
Jul 09 17:41:12.339961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.339772    2576 scope.go:117] "RemoveContainer" containerID="8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62"
Jul 09 17:41:12.347982 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.347958    2576 scope.go:117] "RemoveContainer" containerID="8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62"
Jul 09 17:41:12.348246 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:41:12.348220    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62\": container with ID starting with 8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62 not found: ID does not exist" containerID="8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62"
Jul 09 17:41:12.348315 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.348249    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62"} err="failed to get container status \"8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62\": rpc error: code = NotFound desc = could not find container \"8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62\": container with ID starting with 8f13a269084eb8ed2d36403dca20c43da27d4fea5973a0da92839f8d68f8ce62 not found: ID does not exist"
Jul 09 17:41:12.359319 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.359293    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5698fc6bc8-tsvw9"]
Jul 09 17:41:12.363751 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:12.363734    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5698fc6bc8-tsvw9"]
Jul 09 17:41:13.668985 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:13.668952    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97cbc575-2150-49bf-a52d-49f3877d81a0" path="/var/lib/kubelet/pods/97cbc575-2150-49bf-a52d-49f3877d81a0/volumes"
Jul 09 17:41:14.154478 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:14.154455    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bhf5x"
Jul 09 17:41:28.531320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.531249    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bf9fc9d64-zx59g" podUID="8ee041d1-d800-4605-92e2-4c1a91d7e487" containerName="console" containerID="cri-o://ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9" gracePeriod=15
Jul 09 17:41:28.813414 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.813385    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:41:28.881083 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881057    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-config\") pod \"8ee041d1-d800-4605-92e2-4c1a91d7e487\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") "
Jul 09 17:41:28.881192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-serving-cert\") pod \"8ee041d1-d800-4605-92e2-4c1a91d7e487\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") "
Jul 09 17:41:28.881192 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2bt7\" (UniqueName: \"kubernetes.io/projected/8ee041d1-d800-4605-92e2-4c1a91d7e487-kube-api-access-z2bt7\") pod \"8ee041d1-d800-4605-92e2-4c1a91d7e487\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") "
Jul 09 17:41:28.881275 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881190    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-oauth-config\") pod \"8ee041d1-d800-4605-92e2-4c1a91d7e487\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") "
Jul 09 17:41:28.881275 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881222    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-oauth-serving-cert\") pod \"8ee041d1-d800-4605-92e2-4c1a91d7e487\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") "
Jul 09 17:41:28.881275 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881253    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-trusted-ca-bundle\") pod \"8ee041d1-d800-4605-92e2-4c1a91d7e487\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") "
Jul 09 17:41:28.881401 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-service-ca\") pod \"8ee041d1-d800-4605-92e2-4c1a91d7e487\" (UID: \"8ee041d1-d800-4605-92e2-4c1a91d7e487\") "
Jul 09 17:41:28.881454 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881410    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-config" (OuterVolumeSpecName: "console-config") pod "8ee041d1-d800-4605-92e2-4c1a91d7e487" (UID: "8ee041d1-d800-4605-92e2-4c1a91d7e487"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:28.881733 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881622    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-config\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:28.881733 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881684    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8ee041d1-d800-4605-92e2-4c1a91d7e487" (UID: "8ee041d1-d800-4605-92e2-4c1a91d7e487"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:28.881733 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881685    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8ee041d1-d800-4605-92e2-4c1a91d7e487" (UID: "8ee041d1-d800-4605-92e2-4c1a91d7e487"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:28.881936 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.881770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-service-ca" (OuterVolumeSpecName: "service-ca") pod "8ee041d1-d800-4605-92e2-4c1a91d7e487" (UID: "8ee041d1-d800-4605-92e2-4c1a91d7e487"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:41:28.883378 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.883348    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8ee041d1-d800-4605-92e2-4c1a91d7e487" (UID: "8ee041d1-d800-4605-92e2-4c1a91d7e487"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:41:28.883480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.883439    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee041d1-d800-4605-92e2-4c1a91d7e487-kube-api-access-z2bt7" (OuterVolumeSpecName: "kube-api-access-z2bt7") pod "8ee041d1-d800-4605-92e2-4c1a91d7e487" (UID: "8ee041d1-d800-4605-92e2-4c1a91d7e487"). InnerVolumeSpecName "kube-api-access-z2bt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:41:28.883480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.883449    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8ee041d1-d800-4605-92e2-4c1a91d7e487" (UID: "8ee041d1-d800-4605-92e2-4c1a91d7e487"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:41:28.981915 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.981894    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bt7\" (UniqueName: \"kubernetes.io/projected/8ee041d1-d800-4605-92e2-4c1a91d7e487-kube-api-access-z2bt7\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:28.981915 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.981913    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-oauth-config\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:28.982029 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.981921    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-oauth-serving-cert\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:28.982029 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.981930    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-trusted-ca-bundle\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:28.982029 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.981938    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee041d1-d800-4605-92e2-4c1a91d7e487-service-ca\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:28.982029 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:28.981946    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ee041d1-d800-4605-92e2-4c1a91d7e487-console-serving-cert\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:41:29.407222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.407193    2576 generic.go:356] "Generic (PLEG): container finished" podID="8ee041d1-d800-4605-92e2-4c1a91d7e487" containerID="ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9" exitCode=0
Jul 09 17:41:29.407367 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.407236    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf9fc9d64-zx59g"
Jul 09 17:41:29.407367 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.407278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf9fc9d64-zx59g" event={"ID":"8ee041d1-d800-4605-92e2-4c1a91d7e487","Type":"ContainerDied","Data":"ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9"}
Jul 09 17:41:29.407367 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.407330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf9fc9d64-zx59g" event={"ID":"8ee041d1-d800-4605-92e2-4c1a91d7e487","Type":"ContainerDied","Data":"0e3097075a3be5a5828bc19dd0a5136fac0dee7814bedeff8f4e532b0f3134da"}
Jul 09 17:41:29.407367 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.407357    2576 scope.go:117] "RemoveContainer" containerID="ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9"
Jul 09 17:41:29.415818 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.415802    2576 scope.go:117] "RemoveContainer" containerID="ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9"
Jul 09 17:41:29.416080 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:41:29.416062    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9\": container with ID starting with ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9 not found: ID does not exist" containerID="ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9"
Jul 09 17:41:29.416125 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.416088    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9"} err="failed to get container status \"ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9\": rpc error: code = NotFound desc = could not find container \"ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9\": container with ID starting with ed0d8db133f48a3ec7e91be39d90866bd3bdf4192f66fb8f20883b9a544edda9 not found: ID does not exist"
Jul 09 17:41:29.438105 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.438082    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf9fc9d64-zx59g"]
Jul 09 17:41:29.442940 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.442923    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bf9fc9d64-zx59g"]
Jul 09 17:41:29.668477 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:29.668411    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee041d1-d800-4605-92e2-4c1a91d7e487" path="/var/lib/kubelet/pods/8ee041d1-d800-4605-92e2-4c1a91d7e487/volumes"
Jul 09 17:41:31.511812 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:31.511779    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:41:31.515646 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:31.515626    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5975c47848-c26vd"
Jul 09 17:41:53.625100 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:53.625072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:41:53.643731 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:53.643707    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:41:54.536156 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:41:54.536128    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 17:42:11.463056 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.463013    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5449c784b4-nglgv"]
Jul 09 17:42:11.463587 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.463505    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cbc575-2150-49bf-a52d-49f3877d81a0" containerName="console"
Jul 09 17:42:11.463587 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.463523    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cbc575-2150-49bf-a52d-49f3877d81a0" containerName="console"
Jul 09 17:42:11.463587 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.463557    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee041d1-d800-4605-92e2-4c1a91d7e487" containerName="console"
Jul 09 17:42:11.463587 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.463566    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee041d1-d800-4605-92e2-4c1a91d7e487" containerName="console"
Jul 09 17:42:11.463805 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.463676    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee041d1-d800-4605-92e2-4c1a91d7e487" containerName="console"
Jul 09 17:42:11.463805 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.463693    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="97cbc575-2150-49bf-a52d-49f3877d81a0" containerName="console"
Jul 09 17:42:11.466757 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.466730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.470137 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.470110    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 09 17:42:11.470279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.470111    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lp5px\""
Jul 09 17:42:11.470279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.470181    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 09 17:42:11.470279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.470142    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 09 17:42:11.470279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.470118    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 09 17:42:11.471464 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.471448    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 09 17:42:11.475551 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.475516    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 09 17:42:11.479602 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.479537    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5449c784b4-nglgv"]
Jul 09 17:42:11.627933 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.627902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-metrics-client-ca\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.628117 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.627943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-federate-client-tls\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.628117 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.628005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp6dg\" (UniqueName: \"kubernetes.io/projected/c4d402be-dc3c-482c-83b4-bcfd421ca489-kube-api-access-sp6dg\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.628117 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.628097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-secret-telemeter-client\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.628241 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.628128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-telemeter-client-tls\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.628241 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.628144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.628241 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.628167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-serving-certs-ca-bundle\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.628241 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.628192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.728705 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-secret-telemeter-client\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.728705 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-telemeter-client-tls\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.728705 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.728966 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-serving-certs-ca-bundle\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.728966 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.728966 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-metrics-client-ca\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.728966 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-federate-client-tls\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.729285 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.728976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp6dg\" (UniqueName: \"kubernetes.io/projected/c4d402be-dc3c-482c-83b4-bcfd421ca489-kube-api-access-sp6dg\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.729606 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.729584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-serving-certs-ca-bundle\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.729718 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.729697    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.729885 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.729863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4d402be-dc3c-482c-83b4-bcfd421ca489-metrics-client-ca\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.731246 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.731226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-telemeter-client-tls\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.731329 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.731316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.731632 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.731607    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-secret-telemeter-client\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.731632 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.731621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c4d402be-dc3c-482c-83b4-bcfd421ca489-federate-client-tls\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.738361 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.738341    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp6dg\" (UniqueName: \"kubernetes.io/projected/c4d402be-dc3c-482c-83b4-bcfd421ca489-kube-api-access-sp6dg\") pod \"telemeter-client-5449c784b4-nglgv\" (UID: \"c4d402be-dc3c-482c-83b4-bcfd421ca489\") " pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.777356 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.777335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv"
Jul 09 17:42:11.913468 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:11.913433    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5449c784b4-nglgv"]
Jul 09 17:42:11.916434 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:42:11.916400    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d402be_dc3c_482c_83b4_bcfd421ca489.slice/crio-0c07b7bbc3986f4b944d5483475cd2259b795b01462d417e7f54e7dbc85312b1 WatchSource:0}: Error finding container 0c07b7bbc3986f4b944d5483475cd2259b795b01462d417e7f54e7dbc85312b1: Status 404 returned error can't find the container with id 0c07b7bbc3986f4b944d5483475cd2259b795b01462d417e7f54e7dbc85312b1
Jul 09 17:42:12.589909 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:12.589872    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv" event={"ID":"c4d402be-dc3c-482c-83b4-bcfd421ca489","Type":"ContainerStarted","Data":"0c07b7bbc3986f4b944d5483475cd2259b795b01462d417e7f54e7dbc85312b1"}
Jul 09 17:42:13.595547 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:13.595507    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv" event={"ID":"c4d402be-dc3c-482c-83b4-bcfd421ca489","Type":"ContainerStarted","Data":"a8172172227dc32d483995bac0cfa465940954f130f14f4230985e99fab733bf"}
Jul 09 17:42:14.601695 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:14.601661    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv" event={"ID":"c4d402be-dc3c-482c-83b4-bcfd421ca489","Type":"ContainerStarted","Data":"1f7b6e4fff6221c0966d1e1c55fc4eaa8e626b77902f7df403b7ffe1e8a68996"}
Jul 09 17:42:14.602109 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:14.601725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv" event={"ID":"c4d402be-dc3c-482c-83b4-bcfd421ca489","Type":"ContainerStarted","Data":"c8f32a556c378ebaf029c7a806b32ae894ef90ac7196fb5beddcc76aa8a4c41d"}
Jul 09 17:42:14.625347 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:42:14.625289    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5449c784b4-nglgv" podStartSLOduration=2.023462927 podStartE2EDuration="3.625270534s" totalImagesPullingTime="1.601807607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:42:14.62340316 +0000 UTC m=+164.154180191" watchObservedRunningTime="2026-07-09 17:42:14.625270534 +0000 UTC m=+164.156047598"
Jul 09 17:44:30.878643 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:44:30.878612    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:44:30.883317 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:44:30.883294    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 17:44:30.884467 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:44:30.884451    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:47:47.332309 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.332276    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"]
Jul 09 17:47:47.335711 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.335690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.338540 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.338512    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 17:47:47.338652 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.338516    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-d6zlz\""
Jul 09 17:47:47.338652 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.338570    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 17:47:47.344917 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.344896    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"]
Jul 09 17:47:47.472353 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.472316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eef0152f-654d-4695-9067-71ecaee23b86-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w5j5n\" (UID: \"eef0152f-654d-4695-9067-71ecaee23b86\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.472522 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.472426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9jl\" (UniqueName: \"kubernetes.io/projected/eef0152f-654d-4695-9067-71ecaee23b86-kube-api-access-kg9jl\") pod \"cert-manager-cainjector-559cdd8c87-w5j5n\" (UID: \"eef0152f-654d-4695-9067-71ecaee23b86\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.572955 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.572915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eef0152f-654d-4695-9067-71ecaee23b86-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w5j5n\" (UID: \"eef0152f-654d-4695-9067-71ecaee23b86\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.573148 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.572978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9jl\" (UniqueName: \"kubernetes.io/projected/eef0152f-654d-4695-9067-71ecaee23b86-kube-api-access-kg9jl\") pod \"cert-manager-cainjector-559cdd8c87-w5j5n\" (UID: \"eef0152f-654d-4695-9067-71ecaee23b86\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.585351 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.585290    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eef0152f-654d-4695-9067-71ecaee23b86-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w5j5n\" (UID: \"eef0152f-654d-4695-9067-71ecaee23b86\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.585603 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.585584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9jl\" (UniqueName: \"kubernetes.io/projected/eef0152f-654d-4695-9067-71ecaee23b86-kube-api-access-kg9jl\") pod \"cert-manager-cainjector-559cdd8c87-w5j5n\" (UID: \"eef0152f-654d-4695-9067-71ecaee23b86\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.656191 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.656158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"
Jul 09 17:47:47.794127 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.794099    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n"]
Jul 09 17:47:47.801901 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:47:47.801872    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeef0152f_654d_4695_9067_71ecaee23b86.slice/crio-3c321e66e6ee5f2dbd71d3bbc0e0c7a003f4a09d686bb4bcf8185080594efcf5 WatchSource:0}: Error finding container 3c321e66e6ee5f2dbd71d3bbc0e0c7a003f4a09d686bb4bcf8185080594efcf5: Status 404 returned error can't find the container with id 3c321e66e6ee5f2dbd71d3bbc0e0c7a003f4a09d686bb4bcf8185080594efcf5
Jul 09 17:47:47.803927 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.803908    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:47:47.938729 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:47.938653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n" event={"ID":"eef0152f-654d-4695-9067-71ecaee23b86","Type":"ContainerStarted","Data":"3c321e66e6ee5f2dbd71d3bbc0e0c7a003f4a09d686bb4bcf8185080594efcf5"}
Jul 09 17:47:49.034832 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.034578    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kdqh6"]
Jul 09 17:47:49.038444 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.038422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:49.041684 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.041499    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-m98k9\""
Jul 09 17:47:49.048443 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.047776    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kdqh6"]
Jul 09 17:47:49.187235 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.187198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c2495c6-c6b2-45e3-aa1c-c20994c26ed0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kdqh6\" (UID: \"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:49.187409 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.187269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhvz\" (UniqueName: \"kubernetes.io/projected/4c2495c6-c6b2-45e3-aa1c-c20994c26ed0-kube-api-access-wqhvz\") pod \"cert-manager-webhook-6486ffd997-kdqh6\" (UID: \"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:49.288334 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.288254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c2495c6-c6b2-45e3-aa1c-c20994c26ed0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kdqh6\" (UID: \"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:49.288334 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.288317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhvz\" (UniqueName: \"kubernetes.io/projected/4c2495c6-c6b2-45e3-aa1c-c20994c26ed0-kube-api-access-wqhvz\") pod \"cert-manager-webhook-6486ffd997-kdqh6\" (UID: \"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:49.298053 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.297982    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhvz\" (UniqueName: \"kubernetes.io/projected/4c2495c6-c6b2-45e3-aa1c-c20994c26ed0-kube-api-access-wqhvz\") pod \"cert-manager-webhook-6486ffd997-kdqh6\" (UID: \"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:49.298053 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.298021    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c2495c6-c6b2-45e3-aa1c-c20994c26ed0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kdqh6\" (UID: \"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:49.351774 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:49.351717    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:50.930901 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:50.930882    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kdqh6"]
Jul 09 17:47:50.932957 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:47:50.932928    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2495c6_c6b2_45e3_aa1c_c20994c26ed0.slice/crio-5174e77bd415015d15003f0cedd0450308719c3ca0ebe7a130cef086b5ed6051 WatchSource:0}: Error finding container 5174e77bd415015d15003f0cedd0450308719c3ca0ebe7a130cef086b5ed6051: Status 404 returned error can't find the container with id 5174e77bd415015d15003f0cedd0450308719c3ca0ebe7a130cef086b5ed6051
Jul 09 17:47:50.964086 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:50.964015    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n" event={"ID":"eef0152f-654d-4695-9067-71ecaee23b86","Type":"ContainerStarted","Data":"59b0b60647a066636bf9cfad5c77269b94ba97a2ede6a702a75470f90a67525e"}
Jul 09 17:47:50.965755 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:50.965713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6" event={"ID":"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0","Type":"ContainerStarted","Data":"5174e77bd415015d15003f0cedd0450308719c3ca0ebe7a130cef086b5ed6051"}
Jul 09 17:47:50.991547 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:50.991500    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w5j5n" podStartSLOduration=0.930060307 podStartE2EDuration="3.991484333s" totalImagesPullingTime="3.061424026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:47:50.988864111 +0000 UTC m=+500.519641140" watchObservedRunningTime="2026-07-09 17:47:50.991484333 +0000 UTC m=+500.522261363"
Jul 09 17:47:51.971363 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:51.971329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6" event={"ID":"4c2495c6-c6b2-45e3-aa1c-c20994c26ed0","Type":"ContainerStarted","Data":"5804b23972b4ccca6c5de90baa5ad580a133334fd2e18a55165144f7a62ef5c2"}
Jul 09 17:47:51.971817 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:51.971488    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:47:51.992524 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:51.992485    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6" podStartSLOduration=2.992469687 podStartE2EDuration="2.992469687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:47:51.990967459 +0000 UTC m=+501.521744490" watchObservedRunningTime="2026-07-09 17:47:51.992469687 +0000 UTC m=+501.523246716"
Jul 09 17:47:55.140687 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.140653    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tgnc2"]
Jul 09 17:47:55.144564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.144530    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.147417 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.147400    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-577qc\""
Jul 09 17:47:55.156701 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.156676    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tgnc2"]
Jul 09 17:47:55.245825 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.245776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd4e0c7f-30c7-4e02-b83e-b6254c616fab-bound-sa-token\") pod \"cert-manager-55b4d954d5-tgnc2\" (UID: \"bd4e0c7f-30c7-4e02-b83e-b6254c616fab\") " pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.246074 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.245919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdn4b\" (UniqueName: \"kubernetes.io/projected/bd4e0c7f-30c7-4e02-b83e-b6254c616fab-kube-api-access-tdn4b\") pod \"cert-manager-55b4d954d5-tgnc2\" (UID: \"bd4e0c7f-30c7-4e02-b83e-b6254c616fab\") " pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.346499 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.346465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdn4b\" (UniqueName: \"kubernetes.io/projected/bd4e0c7f-30c7-4e02-b83e-b6254c616fab-kube-api-access-tdn4b\") pod \"cert-manager-55b4d954d5-tgnc2\" (UID: \"bd4e0c7f-30c7-4e02-b83e-b6254c616fab\") " pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.346662 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.346562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd4e0c7f-30c7-4e02-b83e-b6254c616fab-bound-sa-token\") pod \"cert-manager-55b4d954d5-tgnc2\" (UID: \"bd4e0c7f-30c7-4e02-b83e-b6254c616fab\") " pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.356425 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.356398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd4e0c7f-30c7-4e02-b83e-b6254c616fab-bound-sa-token\") pod \"cert-manager-55b4d954d5-tgnc2\" (UID: \"bd4e0c7f-30c7-4e02-b83e-b6254c616fab\") " pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.356567 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.356467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdn4b\" (UniqueName: \"kubernetes.io/projected/bd4e0c7f-30c7-4e02-b83e-b6254c616fab-kube-api-access-tdn4b\") pod \"cert-manager-55b4d954d5-tgnc2\" (UID: \"bd4e0c7f-30c7-4e02-b83e-b6254c616fab\") " pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.453457 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.453378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tgnc2"
Jul 09 17:47:55.585397 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.585372    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tgnc2"]
Jul 09 17:47:55.587589 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:47:55.587563    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd4e0c7f_30c7_4e02_b83e_b6254c616fab.slice/crio-cb96bb7490208cc81685a77886138aec6b5cea9c44fafee5a3b0077aa955c1d9 WatchSource:0}: Error finding container cb96bb7490208cc81685a77886138aec6b5cea9c44fafee5a3b0077aa955c1d9: Status 404 returned error can't find the container with id cb96bb7490208cc81685a77886138aec6b5cea9c44fafee5a3b0077aa955c1d9
Jul 09 17:47:55.989273 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.989240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tgnc2" event={"ID":"bd4e0c7f-30c7-4e02-b83e-b6254c616fab","Type":"ContainerStarted","Data":"6cc258c1acf93c47447be90b4b72bcc01e193166da5cbbef62a75e747e48ec7d"}
Jul 09 17:47:55.989447 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:55.989281    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tgnc2" event={"ID":"bd4e0c7f-30c7-4e02-b83e-b6254c616fab","Type":"ContainerStarted","Data":"cb96bb7490208cc81685a77886138aec6b5cea9c44fafee5a3b0077aa955c1d9"}
Jul 09 17:47:56.008539 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:56.008487    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tgnc2" podStartSLOduration=1.008471802 podStartE2EDuration="1.008471802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:47:56.008222937 +0000 UTC m=+505.538999969" watchObservedRunningTime="2026-07-09 17:47:56.008471802 +0000 UTC m=+505.539248832"
Jul 09 17:47:57.978518 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:47:57.978489    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kdqh6"
Jul 09 17:48:32.147414 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.147378    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"]
Jul 09 17:48:32.149844 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.149829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.152506 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.152481    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 17:48:32.152506 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.152504    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 17:48:32.152667 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.152485    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 17:48:32.153668 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.153652    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:48:32.153718 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.153672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8gxlc\""
Jul 09 17:48:32.153763 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.153724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 17:48:32.165397 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.165375    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"]
Jul 09 17:48:32.288529 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.288501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gc4w\" (UniqueName: \"kubernetes.io/projected/97a21e99-4e62-4b47-b1fb-c330a64780a6-kube-api-access-5gc4w\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.288685 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.288549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/97a21e99-4e62-4b47-b1fb-c330a64780a6-manager-config\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.288685 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.288636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97a21e99-4e62-4b47-b1fb-c330a64780a6-metrics-cert\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.288759 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.288682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97a21e99-4e62-4b47-b1fb-c330a64780a6-cert\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.389582 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.389548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97a21e99-4e62-4b47-b1fb-c330a64780a6-cert\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.389736 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.389601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gc4w\" (UniqueName: \"kubernetes.io/projected/97a21e99-4e62-4b47-b1fb-c330a64780a6-kube-api-access-5gc4w\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.389736 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.389654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/97a21e99-4e62-4b47-b1fb-c330a64780a6-manager-config\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.389736 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.389711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97a21e99-4e62-4b47-b1fb-c330a64780a6-metrics-cert\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.390390 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.390365    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/97a21e99-4e62-4b47-b1fb-c330a64780a6-manager-config\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.392295 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.392273    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97a21e99-4e62-4b47-b1fb-c330a64780a6-cert\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.392379 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.392307    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97a21e99-4e62-4b47-b1fb-c330a64780a6-metrics-cert\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.404085 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.404008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gc4w\" (UniqueName: \"kubernetes.io/projected/97a21e99-4e62-4b47-b1fb-c330a64780a6-kube-api-access-5gc4w\") pod \"lws-controller-manager-6cd758bd69-ld6ng\" (UID: \"97a21e99-4e62-4b47-b1fb-c330a64780a6\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.459534 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.459512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:32.615579 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:32.615557    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"]
Jul 09 17:48:32.617789 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:48:32.617765    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a21e99_4e62_4b47_b1fb_c330a64780a6.slice/crio-7c6d92dfb3edd0eda521e0993c53fc359188db5abfc2009f57da428d38fefba9 WatchSource:0}: Error finding container 7c6d92dfb3edd0eda521e0993c53fc359188db5abfc2009f57da428d38fefba9: Status 404 returned error can't find the container with id 7c6d92dfb3edd0eda521e0993c53fc359188db5abfc2009f57da428d38fefba9
Jul 09 17:48:33.142320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:33.142282    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng" event={"ID":"97a21e99-4e62-4b47-b1fb-c330a64780a6","Type":"ContainerStarted","Data":"7c6d92dfb3edd0eda521e0993c53fc359188db5abfc2009f57da428d38fefba9"}
Jul 09 17:48:36.159222 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:36.159183    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng" event={"ID":"97a21e99-4e62-4b47-b1fb-c330a64780a6","Type":"ContainerStarted","Data":"89709d40a4d1b6e7dc4eb7bb4719a28b0b7adf31c8f343af6bc85d01704fa72e"}
Jul 09 17:48:36.159667 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:36.159310    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:36.179838 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:36.179796    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng" podStartSLOduration=1.3088983619999999 podStartE2EDuration="4.179781299s" totalImagesPullingTime="2.870882937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:48:36.177753655 +0000 UTC m=+545.708530704" watchObservedRunningTime="2026-07-09 17:48:36.179781299 +0000 UTC m=+545.710558382"
Jul 09 17:48:47.165781 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:47.165751    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-ld6ng"
Jul 09 17:48:53.788938 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:53.788908    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"]
Jul 09 17:48:53.792148 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:53.792131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"
Jul 09 17:48:53.796109 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:53.796081    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 17:48:53.796216 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:53.796086    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 17:48:53.797185 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:53.797166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rrj4p\""
Jul 09 17:48:53.812432 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:53.812404    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"]
Jul 09 17:48:53.977555 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:53.977519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6l6c\" (UniqueName: \"kubernetes.io/projected/3aefcf51-79c0-4275-aa20-638cbabd5900-kube-api-access-z6l6c\") pod \"limitador-operator-controller-manager-6f9f468797-xh7g7\" (UID: \"3aefcf51-79c0-4275-aa20-638cbabd5900\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"
Jul 09 17:48:54.078492 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:54.078456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6l6c\" (UniqueName: \"kubernetes.io/projected/3aefcf51-79c0-4275-aa20-638cbabd5900-kube-api-access-z6l6c\") pod \"limitador-operator-controller-manager-6f9f468797-xh7g7\" (UID: \"3aefcf51-79c0-4275-aa20-638cbabd5900\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"
Jul 09 17:48:54.091242 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:54.091211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6l6c\" (UniqueName: \"kubernetes.io/projected/3aefcf51-79c0-4275-aa20-638cbabd5900-kube-api-access-z6l6c\") pod \"limitador-operator-controller-manager-6f9f468797-xh7g7\" (UID: \"3aefcf51-79c0-4275-aa20-638cbabd5900\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"
Jul 09 17:48:54.101875 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:54.101849    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"
Jul 09 17:48:54.245240 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:54.245214    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"]
Jul 09 17:48:54.246921 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:48:54.246894    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aefcf51_79c0_4275_aa20_638cbabd5900.slice/crio-e60fd772342c62df5e3b9fe82378cd34944f85c155db6ba76600c70a61a405b2 WatchSource:0}: Error finding container e60fd772342c62df5e3b9fe82378cd34944f85c155db6ba76600c70a61a405b2: Status 404 returned error can't find the container with id e60fd772342c62df5e3b9fe82378cd34944f85c155db6ba76600c70a61a405b2
Jul 09 17:48:55.240172 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:55.240133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7" event={"ID":"3aefcf51-79c0-4275-aa20-638cbabd5900","Type":"ContainerStarted","Data":"e60fd772342c62df5e3b9fe82378cd34944f85c155db6ba76600c70a61a405b2"}
Jul 09 17:48:57.251629 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:57.251598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7" event={"ID":"3aefcf51-79c0-4275-aa20-638cbabd5900","Type":"ContainerStarted","Data":"fce73b1741542bfa36f06995931d1d2d582343a289ee563ebfdb847c14a354bc"}
Jul 09 17:48:57.251935 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:57.251739    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"
Jul 09 17:48:57.272632 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:48:57.272585    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7" podStartSLOduration=1.366758447 podStartE2EDuration="4.272571352s" totalImagesPullingTime="2.905812905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:48:57.2712182 +0000 UTC m=+566.801995236" watchObservedRunningTime="2026-07-09 17:48:57.272571352 +0000 UTC m=+566.803348381"
Jul 09 17:49:08.258467 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:08.258441    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-xh7g7"
Jul 09 17:49:30.913721 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:30.913693    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:49:30.916742 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:30.916721    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:49:44.020910 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.020873    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-t6fxz"]
Jul 09 17:49:44.027917 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.027899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.030744 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.030721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 17:49:44.031019 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.031002    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6l6kt\""
Jul 09 17:49:44.040130 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.040109    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-t6fxz"]
Jul 09 17:49:44.065695 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.065662    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-t6fxz"]
Jul 09 17:49:44.193304 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.193261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cc22cd2-6536-4fdb-b3bf-adc56315c8a2-config-file\") pod \"limitador-limitador-69574b596d-t6fxz\" (UID: \"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2\") " pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.193472 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.193321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jg7c\" (UniqueName: \"kubernetes.io/projected/5cc22cd2-6536-4fdb-b3bf-adc56315c8a2-kube-api-access-9jg7c\") pod \"limitador-limitador-69574b596d-t6fxz\" (UID: \"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2\") " pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.294403 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.294318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cc22cd2-6536-4fdb-b3bf-adc56315c8a2-config-file\") pod \"limitador-limitador-69574b596d-t6fxz\" (UID: \"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2\") " pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.294543 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.294401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jg7c\" (UniqueName: \"kubernetes.io/projected/5cc22cd2-6536-4fdb-b3bf-adc56315c8a2-kube-api-access-9jg7c\") pod \"limitador-limitador-69574b596d-t6fxz\" (UID: \"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2\") " pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.294969 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.294951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cc22cd2-6536-4fdb-b3bf-adc56315c8a2-config-file\") pod \"limitador-limitador-69574b596d-t6fxz\" (UID: \"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2\") " pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.303983 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.303957    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jg7c\" (UniqueName: \"kubernetes.io/projected/5cc22cd2-6536-4fdb-b3bf-adc56315c8a2-kube-api-access-9jg7c\") pod \"limitador-limitador-69574b596d-t6fxz\" (UID: \"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2\") " pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.337993 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.337955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:44.478669 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:44.478632    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-t6fxz"]
Jul 09 17:49:44.480666 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:49:44.480642    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc22cd2_6536_4fdb_b3bf_adc56315c8a2.slice/crio-ff6be0eef5f2acbe74c0b5dcf05d8c3560f098cded2bcdc78932fb891cc1845d WatchSource:0}: Error finding container ff6be0eef5f2acbe74c0b5dcf05d8c3560f098cded2bcdc78932fb891cc1845d: Status 404 returned error can't find the container with id ff6be0eef5f2acbe74c0b5dcf05d8c3560f098cded2bcdc78932fb891cc1845d
Jul 09 17:49:45.474229 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:45.474191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz" event={"ID":"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2","Type":"ContainerStarted","Data":"ff6be0eef5f2acbe74c0b5dcf05d8c3560f098cded2bcdc78932fb891cc1845d"}
Jul 09 17:49:49.495597 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:49.495561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz" event={"ID":"5cc22cd2-6536-4fdb-b3bf-adc56315c8a2","Type":"ContainerStarted","Data":"59dd8e3c07694edea1f150be7ea2e4fd06f9f1873ba7d79a366f6af114a099fd"}
Jul 09 17:49:49.496034 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:49.495638    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:49:49.527774 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:49:49.527723    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz" podStartSLOduration=2.502563437 podStartE2EDuration="6.527707071s" totalImagesPullingTime="4.025143634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:49:49.526620543 +0000 UTC m=+619.057397574" watchObservedRunningTime="2026-07-09 17:49:49.527707071 +0000 UTC m=+619.058484101"
Jul 09 17:50:00.501539 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:50:00.501461    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-t6fxz"
Jul 09 17:53:11.351020 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.350991    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"]
Jul 09 17:53:11.354565 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.354548    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.357576 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.357552    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:53:11.357712 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.357556    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 17:53:11.357712 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.357633    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fg56t\""
Jul 09 17:53:11.357712 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.357554    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 17:53:11.358616 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.358587    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 09 17:53:11.366824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.366804    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"]
Jul 09 17:53:11.490742 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.490722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-dshm\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.490861 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.490750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.490861 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.490782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-tmp-dir\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.490861 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.490814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqclf\" (UniqueName: \"kubernetes.io/projected/f035ec53-9130-4d78-afcc-62b346eaae85-kube-api-access-xqclf\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.490985 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.490952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f035ec53-9130-4d78-afcc-62b346eaae85-tls-certs\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.491027 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.491012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-home\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.491089 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.491032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-model-cache\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592191 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-home\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592293 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-model-cache\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592293 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-dshm\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592293 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592293 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-tmp-dir\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592518 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqclf\" (UniqueName: \"kubernetes.io/projected/f035ec53-9130-4d78-afcc-62b346eaae85-kube-api-access-xqclf\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592518 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f035ec53-9130-4d78-afcc-62b346eaae85-tls-certs\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592614 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-model-cache\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592614 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-home\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592723 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-tmp-dir\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.592816 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.592796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.594468 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.594451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-dshm\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.594734 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.594715    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f035ec53-9130-4d78-afcc-62b346eaae85-tls-certs\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.600406 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.600384    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqclf\" (UniqueName: \"kubernetes.io/projected/f035ec53-9130-4d78-afcc-62b346eaae85-kube-api-access-xqclf\") pod \"precise-prefix-cache-test-kserve-56cbd96798-gjgcf\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.664920 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.664852    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:11.801878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.801857    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"]
Jul 09 17:53:11.803467 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:53:11.803443    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf035ec53_9130_4d78_afcc_62b346eaae85.slice/crio-6ad9418dbdceb20d64be207d03cd7b65acb99c31d2519d06cecc7d645c3817e5 WatchSource:0}: Error finding container 6ad9418dbdceb20d64be207d03cd7b65acb99c31d2519d06cecc7d645c3817e5: Status 404 returned error can't find the container with id 6ad9418dbdceb20d64be207d03cd7b65acb99c31d2519d06cecc7d645c3817e5
Jul 09 17:53:11.805215 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:11.805196    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:53:12.447622 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:12.447581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" event={"ID":"f035ec53-9130-4d78-afcc-62b346eaae85","Type":"ContainerStarted","Data":"6ad9418dbdceb20d64be207d03cd7b65acb99c31d2519d06cecc7d645c3817e5"}
Jul 09 17:53:16.473573 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:16.473534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" event={"ID":"f035ec53-9130-4d78-afcc-62b346eaae85","Type":"ContainerStarted","Data":"0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab"}
Jul 09 17:53:19.490300 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:19.490267    2576 generic.go:356] "Generic (PLEG): container finished" podID="f035ec53-9130-4d78-afcc-62b346eaae85" containerID="0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab" exitCode=0
Jul 09 17:53:19.490668 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:19.490344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" event={"ID":"f035ec53-9130-4d78-afcc-62b346eaae85","Type":"ContainerDied","Data":"0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab"}
Jul 09 17:53:20.180877 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.180841    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"]
Jul 09 17:53:20.184466 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.184444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.187518 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.187496    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 09 17:53:20.195825 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.195769    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"]
Jul 09 17:53:20.270798 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.270769    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.270952 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.270840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82pc\" (UniqueName: \"kubernetes.io/projected/61b97f74-d699-4a44-b45b-bd8c55e1a897-kube-api-access-r82pc\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.270952 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.270922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.271084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.270961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.271084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.270981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b97f74-d699-4a44-b45b-bd8c55e1a897-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.271084 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.271074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372275 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372433 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82pc\" (UniqueName: \"kubernetes.io/projected/61b97f74-d699-4a44-b45b-bd8c55e1a897-kube-api-access-r82pc\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372433 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372433 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372598 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b97f74-d699-4a44-b45b-bd8c55e1a897-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372598 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372711 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372837 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.372837 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.372769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.374976 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.374953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.375303 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.375287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b97f74-d699-4a44-b45b-bd8c55e1a897-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.381979 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.381959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82pc\" (UniqueName: \"kubernetes.io/projected/61b97f74-d699-4a44-b45b-bd8c55e1a897-kube-api-access-r82pc\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.499665 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.499592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:20.652871 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:20.652837    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"]
Jul 09 17:53:20.656710 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:53:20.656683    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b97f74_d699_4a44_b45b_bd8c55e1a897.slice/crio-ce74d5d1f57524d451d6d7db3185a6431c1846794bab4ad12cecd30326187f9d WatchSource:0}: Error finding container ce74d5d1f57524d451d6d7db3185a6431c1846794bab4ad12cecd30326187f9d: Status 404 returned error can't find the container with id ce74d5d1f57524d451d6d7db3185a6431c1846794bab4ad12cecd30326187f9d
Jul 09 17:53:21.501502 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.501467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv" event={"ID":"61b97f74-d699-4a44-b45b-bd8c55e1a897","Type":"ContainerStarted","Data":"bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5"}
Jul 09 17:53:21.501899 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.501534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv" event={"ID":"61b97f74-d699-4a44-b45b-bd8c55e1a897","Type":"ContainerStarted","Data":"ce74d5d1f57524d451d6d7db3185a6431c1846794bab4ad12cecd30326187f9d"}
Jul 09 17:53:21.503229 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.503212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" event={"ID":"f035ec53-9130-4d78-afcc-62b346eaae85","Type":"ContainerStarted","Data":"f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6"}
Jul 09 17:53:21.522480 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.522439    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv" podStartSLOduration=1.522425398 podStartE2EDuration="1.522425398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:53:21.522343644 +0000 UTC m=+831.053120671" watchObservedRunningTime="2026-07-09 17:53:21.522425398 +0000 UTC m=+831.053202428"
Jul 09 17:53:21.543660 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.543623    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" podStartSLOduration=2.11387048 podStartE2EDuration="10.543613284s" totalImagesPullingTime="4.705339196s" totalInitContainerRuntime="3.724403608s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:53:21.542767761 +0000 UTC m=+831.073544801" watchObservedRunningTime="2026-07-09 17:53:21.543613284 +0000 UTC m=+831.074390313"
Jul 09 17:53:21.669735 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.669712    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:21.669868 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.669757    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:21.678247 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:21.678229    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:22.500059 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:22.500015    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:22.519298 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:22.519272    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:53:30.500729 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:30.500686    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:30.512808 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:30.512788    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:30.557533 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:30.557511    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:32.111254 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:32.111226    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"]
Jul 09 17:53:46.095308 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.095279    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"]
Jul 09 17:53:46.108263 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.108237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.111072 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.111053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 09 17:53:46.119607 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.119583    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"]
Jul 09 17:53:46.202391 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.202371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tls-certs\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.202498 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.202406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-model-cache\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.202498 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.202435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tmp-dir\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.202498 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.202458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kserve-provision-location\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.202498 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.202496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-dshm\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.202625 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.202511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-home\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.202625 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.202552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wz8r\" (UniqueName: \"kubernetes.io/projected/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kube-api-access-9wz8r\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303585 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tmp-dir\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303684 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kserve-provision-location\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303684 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-dshm\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303684 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-home\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303684 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wz8r\" (UniqueName: \"kubernetes.io/projected/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kube-api-access-9wz8r\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303882 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tls-certs\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303882 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-model-cache\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.303985 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.303962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kserve-provision-location\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.304074 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.304018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tmp-dir\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.304161 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.304139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-model-cache\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.304204 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.304170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-home\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.305780 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.305759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-dshm\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.306157 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.306140    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tls-certs\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.313546 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.313524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wz8r\" (UniqueName: \"kubernetes.io/projected/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kube-api-access-9wz8r\") pod \"auth-enabled-test-kserve-7d99555666-77kds\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.419190 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.419129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:53:46.556854 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.556755    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"]
Jul 09 17:53:46.558325 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:53:46.558195    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf5ea4a_889c_4c1d_ab6f_d885fe5b6e7c.slice/crio-1ea707c18462fd35ed10a439428b2c6084e830f2e49f3f9f7d725563ab5cdccc WatchSource:0}: Error finding container 1ea707c18462fd35ed10a439428b2c6084e830f2e49f3f9f7d725563ab5cdccc: Status 404 returned error can't find the container with id 1ea707c18462fd35ed10a439428b2c6084e830f2e49f3f9f7d725563ab5cdccc
Jul 09 17:53:46.622616 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.622594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" event={"ID":"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c","Type":"ContainerStarted","Data":"ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46"}
Jul 09 17:53:46.622708 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:46.622648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" event={"ID":"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c","Type":"ContainerStarted","Data":"1ea707c18462fd35ed10a439428b2c6084e830f2e49f3f9f7d725563ab5cdccc"}
Jul 09 17:53:47.578324 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:47.577912    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv" podUID="61b97f74-d699-4a44-b45b-bd8c55e1a897" containerName="main" containerID="cri-o://bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5" gracePeriod=45
Jul 09 17:53:47.867429 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:47.867405    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:48.021720 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.021685    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-dshm\") pod \"61b97f74-d699-4a44-b45b-bd8c55e1a897\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") "
Jul 09 17:53:48.021893 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.021731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-model-cache\") pod \"61b97f74-d699-4a44-b45b-bd8c55e1a897\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") "
Jul 09 17:53:48.021893 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.021815    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-tmp-dir\") pod \"61b97f74-d699-4a44-b45b-bd8c55e1a897\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") "
Jul 09 17:53:48.021893 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.021852    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b97f74-d699-4a44-b45b-bd8c55e1a897-tls-certs\") pod \"61b97f74-d699-4a44-b45b-bd8c55e1a897\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") "
Jul 09 17:53:48.021893 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.021868    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-home\") pod \"61b97f74-d699-4a44-b45b-bd8c55e1a897\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") "
Jul 09 17:53:48.022146 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.021899    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r82pc\" (UniqueName: \"kubernetes.io/projected/61b97f74-d699-4a44-b45b-bd8c55e1a897-kube-api-access-r82pc\") pod \"61b97f74-d699-4a44-b45b-bd8c55e1a897\" (UID: \"61b97f74-d699-4a44-b45b-bd8c55e1a897\") "
Jul 09 17:53:48.022146 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.022008    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-model-cache" (OuterVolumeSpecName: "model-cache") pod "61b97f74-d699-4a44-b45b-bd8c55e1a897" (UID: "61b97f74-d699-4a44-b45b-bd8c55e1a897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.022146 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.022091    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61b97f74-d699-4a44-b45b-bd8c55e1a897" (UID: "61b97f74-d699-4a44-b45b-bd8c55e1a897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.022290 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.022160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-home" (OuterVolumeSpecName: "home") pod "61b97f74-d699-4a44-b45b-bd8c55e1a897" (UID: "61b97f74-d699-4a44-b45b-bd8c55e1a897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.022337 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.022290    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.022337 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.022308    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.022337 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.022320    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.024135 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.024112    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-dshm" (OuterVolumeSpecName: "dshm") pod "61b97f74-d699-4a44-b45b-bd8c55e1a897" (UID: "61b97f74-d699-4a44-b45b-bd8c55e1a897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.024259 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.024161    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b97f74-d699-4a44-b45b-bd8c55e1a897-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61b97f74-d699-4a44-b45b-bd8c55e1a897" (UID: "61b97f74-d699-4a44-b45b-bd8c55e1a897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:53:48.024259 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.024185    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b97f74-d699-4a44-b45b-bd8c55e1a897-kube-api-access-r82pc" (OuterVolumeSpecName: "kube-api-access-r82pc") pod "61b97f74-d699-4a44-b45b-bd8c55e1a897" (UID: "61b97f74-d699-4a44-b45b-bd8c55e1a897"). InnerVolumeSpecName "kube-api-access-r82pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:53:48.123564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.123516    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b97f74-d699-4a44-b45b-bd8c55e1a897-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.123564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.123536    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r82pc\" (UniqueName: \"kubernetes.io/projected/61b97f74-d699-4a44-b45b-bd8c55e1a897-kube-api-access-r82pc\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.123564 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.123545    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b97f74-d699-4a44-b45b-bd8c55e1a897-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.639561 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.639526    2576 generic.go:356] "Generic (PLEG): container finished" podID="61b97f74-d699-4a44-b45b-bd8c55e1a897" containerID="bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5" exitCode=0
Jul 09 17:53:48.640004 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.639596    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"
Jul 09 17:53:48.640004 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.639626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv" event={"ID":"61b97f74-d699-4a44-b45b-bd8c55e1a897","Type":"ContainerDied","Data":"bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5"}
Jul 09 17:53:48.640004 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.639680    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv" event={"ID":"61b97f74-d699-4a44-b45b-bd8c55e1a897","Type":"ContainerDied","Data":"ce74d5d1f57524d451d6d7db3185a6431c1846794bab4ad12cecd30326187f9d"}
Jul 09 17:53:48.640004 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.639715    2576 scope.go:117] "RemoveContainer" containerID="bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5"
Jul 09 17:53:48.653146 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.653127    2576 scope.go:117] "RemoveContainer" containerID="bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5"
Jul 09 17:53:48.653489 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:53:48.653470    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5\": container with ID starting with bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5 not found: ID does not exist" containerID="bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5"
Jul 09 17:53:48.653560 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.653500    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5"} err="failed to get container status \"bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5\": rpc error: code = NotFound desc = could not find container \"bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5\": container with ID starting with bda1d8eeec5f61e62e8c27a97438ca207257b2a804c32ca5e8d4cdabc51771c5 not found: ID does not exist"
Jul 09 17:53:48.678170 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.678146    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"]
Jul 09 17:53:48.684737 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:48.684712    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv"]
Jul 09 17:53:49.670398 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:49.670365    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b97f74-d699-4a44-b45b-bd8c55e1a897" path="/var/lib/kubelet/pods/61b97f74-d699-4a44-b45b-bd8c55e1a897/volumes"
Jul 09 17:53:50.652365 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:50.652333    2576 generic.go:356] "Generic (PLEG): container finished" podID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerID="ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46" exitCode=0
Jul 09 17:53:50.652536 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:53:50.652405    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" event={"ID":"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c","Type":"ContainerDied","Data":"ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46"}
Jul 09 17:54:17.834099 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:17.834060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" event={"ID":"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c","Type":"ContainerStarted","Data":"7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748"}
Jul 09 17:54:17.861517 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:17.861469    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podStartSLOduration=1.838217163 podStartE2EDuration="31.861453155s" totalImagesPullingTime="26.831292257s" totalInitContainerRuntime="3.191943735s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:54:17.859283849 +0000 UTC m=+887.390060896" watchObservedRunningTime="2026-07-09 17:54:17.861453155 +0000 UTC m=+887.392230184"
Jul 09 17:54:18.419805 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:18.419773    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:54:24.616085 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:24.615984    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"]
Jul 09 17:54:26.419973 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:26.419935    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:54:26.421264 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:26.421229    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:54:30.948898 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:30.948868    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:54:30.949866 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:30.949846    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:54:36.419930 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:36.419877    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:54:39.636335 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.636294    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" podUID="f035ec53-9130-4d78-afcc-62b346eaae85" containerName="main" containerID="cri-o://f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6" gracePeriod=45
Jul 09 17:54:39.886941 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.886883    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:54:39.928920 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.928881    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-dshm\") pod \"f035ec53-9130-4d78-afcc-62b346eaae85\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") "
Jul 09 17:54:39.929087 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.928929    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-home\") pod \"f035ec53-9130-4d78-afcc-62b346eaae85\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") "
Jul 09 17:54:39.929087 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.928971    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-tmp-dir\") pod \"f035ec53-9130-4d78-afcc-62b346eaae85\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") "
Jul 09 17:54:39.929087 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.928999    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f035ec53-9130-4d78-afcc-62b346eaae85-tls-certs\") pod \"f035ec53-9130-4d78-afcc-62b346eaae85\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") "
Jul 09 17:54:39.929087 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929080    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqclf\" (UniqueName: \"kubernetes.io/projected/f035ec53-9130-4d78-afcc-62b346eaae85-kube-api-access-xqclf\") pod \"f035ec53-9130-4d78-afcc-62b346eaae85\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") "
Jul 09 17:54:39.929314 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929121    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-model-cache\") pod \"f035ec53-9130-4d78-afcc-62b346eaae85\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") "
Jul 09 17:54:39.929314 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929225    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f035ec53-9130-4d78-afcc-62b346eaae85" (UID: "f035ec53-9130-4d78-afcc-62b346eaae85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.929314 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929257    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-home" (OuterVolumeSpecName: "home") pod "f035ec53-9130-4d78-afcc-62b346eaae85" (UID: "f035ec53-9130-4d78-afcc-62b346eaae85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.929531 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929476    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-model-cache" (OuterVolumeSpecName: "model-cache") pod "f035ec53-9130-4d78-afcc-62b346eaae85" (UID: "f035ec53-9130-4d78-afcc-62b346eaae85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.929657 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929558    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-kserve-provision-location\") pod \"f035ec53-9130-4d78-afcc-62b346eaae85\" (UID: \"f035ec53-9130-4d78-afcc-62b346eaae85\") "
Jul 09 17:54:39.929880 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929860    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:54:39.929956 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929888    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:54:39.929956 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.929901    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:54:39.931288 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.931263    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-dshm" (OuterVolumeSpecName: "dshm") pod "f035ec53-9130-4d78-afcc-62b346eaae85" (UID: "f035ec53-9130-4d78-afcc-62b346eaae85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.931638 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.931611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f035ec53-9130-4d78-afcc-62b346eaae85-kube-api-access-xqclf" (OuterVolumeSpecName: "kube-api-access-xqclf") pod "f035ec53-9130-4d78-afcc-62b346eaae85" (UID: "f035ec53-9130-4d78-afcc-62b346eaae85"). InnerVolumeSpecName "kube-api-access-xqclf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:54:39.931824 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.931804    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f035ec53-9130-4d78-afcc-62b346eaae85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f035ec53-9130-4d78-afcc-62b346eaae85" (UID: "f035ec53-9130-4d78-afcc-62b346eaae85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:54:39.949893 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.949867    2576 generic.go:356] "Generic (PLEG): container finished" podID="f035ec53-9130-4d78-afcc-62b346eaae85" containerID="f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6" exitCode=0
Jul 09 17:54:39.950015 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.949934    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"
Jul 09 17:54:39.950015 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.949949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" event={"ID":"f035ec53-9130-4d78-afcc-62b346eaae85","Type":"ContainerDied","Data":"f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6"}
Jul 09 17:54:39.950015 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.950007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf" event={"ID":"f035ec53-9130-4d78-afcc-62b346eaae85","Type":"ContainerDied","Data":"6ad9418dbdceb20d64be207d03cd7b65acb99c31d2519d06cecc7d645c3817e5"}
Jul 09 17:54:39.950204 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.950030    2576 scope.go:117] "RemoveContainer" containerID="f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6"
Jul 09 17:54:39.958570 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.958552    2576 scope.go:117] "RemoveContainer" containerID="0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab"
Jul 09 17:54:39.966363 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.966342    2576 scope.go:117] "RemoveContainer" containerID="f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6"
Jul 09 17:54:39.966629 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:54:39.966606    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6\": container with ID starting with f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6 not found: ID does not exist" containerID="f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6"
Jul 09 17:54:39.966706 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.966640    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6"} err="failed to get container status \"f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6\": rpc error: code = NotFound desc = could not find container \"f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6\": container with ID starting with f26407df61556f1b3036aef64697847629ec86b959da795d4af6f100009106f6 not found: ID does not exist"
Jul 09 17:54:39.966706 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.966664    2576 scope.go:117] "RemoveContainer" containerID="0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab"
Jul 09 17:54:39.966928 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:54:39.966911    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab\": container with ID starting with 0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab not found: ID does not exist" containerID="0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab"
Jul 09 17:54:39.967006 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:39.966934    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab"} err="failed to get container status \"0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab\": rpc error: code = NotFound desc = could not find container \"0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab\": container with ID starting with 0c29451dd038be758c58a4441592b6cc836203ef619260607cf34cdc732b16ab not found: ID does not exist"
Jul 09 17:54:40.010205 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:40.010160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f035ec53-9130-4d78-afcc-62b346eaae85" (UID: "f035ec53-9130-4d78-afcc-62b346eaae85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:40.030435 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:40.030406    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqclf\" (UniqueName: \"kubernetes.io/projected/f035ec53-9130-4d78-afcc-62b346eaae85-kube-api-access-xqclf\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.030435 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:40.030431    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.030435 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:40.030441    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f035ec53-9130-4d78-afcc-62b346eaae85-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.030623 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:40.030453    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f035ec53-9130-4d78-afcc-62b346eaae85-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.274478 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:40.274449    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"]
Jul 09 17:54:40.276911 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:40.276887    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf"]
Jul 09 17:54:41.671501 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:41.671465    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f035ec53-9130-4d78-afcc-62b346eaae85" path="/var/lib/kubelet/pods/f035ec53-9130-4d78-afcc-62b346eaae85/volumes"
Jul 09 17:54:46.420406 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:46.420363    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:54:56.420579 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:54:56.420528    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:55:06.419767 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:55:06.419719    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:55:16.420249 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:55:16.420200    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:55:26.420333 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:55:26.420283    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:55:36.420399 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:55:36.420357    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:55:46.419900 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:55:46.419858    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:55:56.420437 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:55:56.420345    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:56:06.429478 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:06.429451    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:56:06.436923 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:06.436892    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:56:32.070426 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:32.070387    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"]
Jul 09 17:56:47.093956 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.093898    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" containerID="cri-o://7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748" gracePeriod=45
Jul 09 17:56:47.430749 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.430650    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 09 17:56:47.853165 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.853143    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:56:47.949349 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949322    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tls-certs\") pod \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") "
Jul 09 17:56:47.949482 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949363    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kserve-provision-location\") pod \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") "
Jul 09 17:56:47.949482 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949387    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-dshm\") pod \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") "
Jul 09 17:56:47.949482 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949419    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-model-cache\") pod \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") "
Jul 09 17:56:47.949654 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949520    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tmp-dir\") pod \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") "
Jul 09 17:56:47.949654 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949544    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wz8r\" (UniqueName: \"kubernetes.io/projected/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kube-api-access-9wz8r\") pod \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") "
Jul 09 17:56:47.949654 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949603    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-home\") pod \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\" (UID: \"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c\") "
Jul 09 17:56:47.949819 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" (UID: "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:56:47.950015 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.949985    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:56:47.950440 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.950416    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-home" (OuterVolumeSpecName: "home") pod "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" (UID: "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:56:47.951559 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.951533    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" (UID: "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:56:47.951902 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.951872    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-dshm" (OuterVolumeSpecName: "dshm") pod "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" (UID: "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:56:47.951987 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.951893    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kube-api-access-9wz8r" (OuterVolumeSpecName: "kube-api-access-9wz8r") pod "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" (UID: "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c"). InnerVolumeSpecName "kube-api-access-9wz8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:56:47.961667 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:47.961601    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" (UID: "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:56:48.005711 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.005687    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" (UID: "acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:56:48.051203 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.051181    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:56:48.051203 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.051200    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:56:48.051318 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.051211    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:56:48.051318 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.051219    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:56:48.051318 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.051228    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:56:48.051318 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.051236    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wz8r\" (UniqueName: \"kubernetes.io/projected/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c-kube-api-access-9wz8r\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:56:48.592976 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.592940    2576 generic.go:356] "Generic (PLEG): container finished" podID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerID="7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748" exitCode=0
Jul 09 17:56:48.593536 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.593510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" event={"ID":"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c","Type":"ContainerDied","Data":"7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748"}
Jul 09 17:56:48.593722 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.593705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds" event={"ID":"acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c","Type":"ContainerDied","Data":"1ea707c18462fd35ed10a439428b2c6084e830f2e49f3f9f7d725563ab5cdccc"}
Jul 09 17:56:48.593875 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.593841    2576 scope.go:117] "RemoveContainer" containerID="7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748"
Jul 09 17:56:48.593987 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.593924    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"
Jul 09 17:56:48.604107 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.604088    2576 scope.go:117] "RemoveContainer" containerID="ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46"
Jul 09 17:56:48.611823 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.611811    2576 scope.go:117] "RemoveContainer" containerID="7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748"
Jul 09 17:56:48.612053 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:56:48.612020    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748\": container with ID starting with 7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748 not found: ID does not exist" containerID="7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748"
Jul 09 17:56:48.612111 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.612055    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748"} err="failed to get container status \"7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748\": rpc error: code = NotFound desc = could not find container \"7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748\": container with ID starting with 7d055fd0627c334e7d0a05314a7a38dc93d1ef6452cc02f58442f0558e8d0748 not found: ID does not exist"
Jul 09 17:56:48.612111 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.612071    2576 scope.go:117] "RemoveContainer" containerID="ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46"
Jul 09 17:56:48.612301 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:56:48.612283    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46\": container with ID starting with ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46 not found: ID does not exist" containerID="ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46"
Jul 09 17:56:48.612356 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.612311    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46"} err="failed to get container status \"ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46\": rpc error: code = NotFound desc = could not find container \"ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46\": container with ID starting with ffffc27a1c420d921404cc77cb51629943a5767a7c3e7724d02c50f8532abb46 not found: ID does not exist"
Jul 09 17:56:48.618277 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.618259    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"]
Jul 09 17:56:48.623890 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:48.623870    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds"]
Jul 09 17:56:49.668950 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:49.668922    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" path="/var/lib/kubelet/pods/acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c/volumes"
Jul 09 17:56:55.287456 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287427    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"]
Jul 09 17:56:55.287801 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287787    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b97f74-d699-4a44-b45b-bd8c55e1a897" containerName="main"
Jul 09 17:56:55.287801 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287798    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b97f74-d699-4a44-b45b-bd8c55e1a897" containerName="main"
Jul 09 17:56:55.287870 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287812    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="storage-initializer"
Jul 09 17:56:55.287870 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287818    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="storage-initializer"
Jul 09 17:56:55.287870 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287831    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f035ec53-9130-4d78-afcc-62b346eaae85" containerName="storage-initializer"
Jul 09 17:56:55.287870 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287836    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f035ec53-9130-4d78-afcc-62b346eaae85" containerName="storage-initializer"
Jul 09 17:56:55.287870 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287848    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f035ec53-9130-4d78-afcc-62b346eaae85" containerName="main"
Jul 09 17:56:55.287870 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287853    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f035ec53-9130-4d78-afcc-62b346eaae85" containerName="main"
Jul 09 17:56:55.288279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287874    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main"
Jul 09 17:56:55.288279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287879    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main"
Jul 09 17:56:55.288279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287932    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b97f74-d699-4a44-b45b-bd8c55e1a897" containerName="main"
Jul 09 17:56:55.288279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287941    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f035ec53-9130-4d78-afcc-62b346eaae85" containerName="main"
Jul 09 17:56:55.288279 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.287947    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf5ea4a-889c-4c1d-ab6f-d885fe5b6e7c" containerName="main"
Jul 09 17:56:55.291139 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.291123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.293728 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.293703    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 17:56:55.294771 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.294752    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 17:56:55.294865 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.294827    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 09 17:56:55.295000 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.294976    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fg56t\""
Jul 09 17:56:55.300103 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.300082    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"]
Jul 09 17:56:55.412678 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.412652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.412804 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.412681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.412804 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.412722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.412804 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.412778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.412966 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.412867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.412966 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.412923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq6d4\" (UniqueName: \"kubernetes.io/projected/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-kube-api-access-tq6d4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514161 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514267 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514267 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514267 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq6d4\" (UniqueName: \"kubernetes.io/projected/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-kube-api-access-tq6d4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514402 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514402 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514629 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514608    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.514690 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.514659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.516225 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.516202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.516526 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.516509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.521758 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.521738    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq6d4\" (UniqueName: \"kubernetes.io/projected/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-kube-api-access-tq6d4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.602136 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.602113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:56:55.737061 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:55.737021    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"]
Jul 09 17:56:55.739010 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:56:55.738985    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e130fb2_82d4_47b7_aa6f_f835f152b3a6.slice/crio-d486fbe5ae2b0b8acac6c585680b360ec11b25f08b08e91130caa26e383e01cf WatchSource:0}: Error finding container d486fbe5ae2b0b8acac6c585680b360ec11b25f08b08e91130caa26e383e01cf: Status 404 returned error can't find the container with id d486fbe5ae2b0b8acac6c585680b360ec11b25f08b08e91130caa26e383e01cf
Jul 09 17:56:56.632501 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:56.632469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7" event={"ID":"5e130fb2-82d4-47b7-aa6f-f835f152b3a6","Type":"ContainerStarted","Data":"463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3"}
Jul 09 17:56:56.632933 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:56.632515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7" event={"ID":"5e130fb2-82d4-47b7-aa6f-f835f152b3a6","Type":"ContainerStarted","Data":"d486fbe5ae2b0b8acac6c585680b360ec11b25f08b08e91130caa26e383e01cf"}
Jul 09 17:56:56.650644 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:56.650599    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7" podStartSLOduration=1.6505638249999999 podStartE2EDuration="1.650563825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:56:56.649145669 +0000 UTC m=+1046.179922713" watchObservedRunningTime="2026-07-09 17:56:56.650563825 +0000 UTC m=+1046.181340855"
Jul 09 17:56:57.602447 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:56:57.602417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:57:05.602936 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:05.602906    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:57:05.615009 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:05.614991    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:57:05.688434 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:05.688407    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:57:19.781574 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.781546    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"]
Jul 09 17:57:19.785235 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.785214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:19.787687 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.787666    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 09 17:57:19.787939 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.787922    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:57:19.795835 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.795814    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"]
Jul 09 17:57:19.914210 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.914185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-tmp-dir\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:19.914332 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.914214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-model-cache\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:19.914332 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.914237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-dshm\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:19.914332 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.914252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecac386-06ac-4508-b7de-d965e6ff07d0-tls-certs\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:19.914469 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.914363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-home\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:19.914469 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.914395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88fvn\" (UniqueName: \"kubernetes.io/projected/6ecac386-06ac-4508-b7de-d965e6ff07d0-kube-api-access-88fvn\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:19.914469 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:19.914419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015340 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-tmp-dir\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015514 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-model-cache\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015514 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-dshm\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015514 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecac386-06ac-4508-b7de-d965e6ff07d0-tls-certs\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015514 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-home\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015733 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88fvn\" (UniqueName: \"kubernetes.io/projected/6ecac386-06ac-4508-b7de-d965e6ff07d0-kube-api-access-88fvn\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015733 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015834 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015792    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-model-cache\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015834 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-home\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015976 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015957    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-tmp-dir\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.015976 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.015967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.018034 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.018014    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-dshm\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.018271 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.018254    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecac386-06ac-4508-b7de-d965e6ff07d0-tls-certs\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.024239 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.024216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88fvn\" (UniqueName: \"kubernetes.io/projected/6ecac386-06ac-4508-b7de-d965e6ff07d0-kube-api-access-88fvn\") pod \"auth-invalid-token-test-kserve-6c6c746446-f2ppq\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.096256 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.096233    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:20.237625 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.237597    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"]
Jul 09 17:57:20.240801 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:57:20.240768    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecac386_06ac_4508_b7de_d965e6ff07d0.slice/crio-14a7df39986e3d46f3a479d877b8eeb9175cbc2ffea16533a0e10572bbff2a00 WatchSource:0}: Error finding container 14a7df39986e3d46f3a479d877b8eeb9175cbc2ffea16533a0e10572bbff2a00: Status 404 returned error can't find the container with id 14a7df39986e3d46f3a479d877b8eeb9175cbc2ffea16533a0e10572bbff2a00
Jul 09 17:57:20.744623 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.744586    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" event={"ID":"6ecac386-06ac-4508-b7de-d965e6ff07d0","Type":"ContainerStarted","Data":"1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9"}
Jul 09 17:57:20.744782 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:20.744646    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" event={"ID":"6ecac386-06ac-4508-b7de-d965e6ff07d0","Type":"ContainerStarted","Data":"14a7df39986e3d46f3a479d877b8eeb9175cbc2ffea16533a0e10572bbff2a00"}
Jul 09 17:57:23.770530 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:23.770448    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerID="1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9" exitCode=0
Jul 09 17:57:23.770530 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:23.770488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" event={"ID":"6ecac386-06ac-4508-b7de-d965e6ff07d0","Type":"ContainerDied","Data":"1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9"}
Jul 09 17:57:24.777850 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:24.777815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" event={"ID":"6ecac386-06ac-4508-b7de-d965e6ff07d0","Type":"ContainerStarted","Data":"352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126"}
Jul 09 17:57:24.802867 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:24.802812    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podStartSLOduration=2.775076946 podStartE2EDuration="5.80279632s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.027719374s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:57:24.800469102 +0000 UTC m=+1074.331246158" watchObservedRunningTime="2026-07-09 17:57:24.80279632 +0000 UTC m=+1074.333573349"
Jul 09 17:57:25.096974 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:25.096932    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:30.097405 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:30.097371    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:57:30.098813 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:30.098790    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:57:40.096611 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:40.096559    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:57:42.011535 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:42.011502    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"]
Jul 09 17:57:50.097362 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:50.097320    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:57:57.032105 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.032067    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7" podUID="5e130fb2-82d4-47b7-aa6f-f835f152b3a6" containerName="main" containerID="cri-o://463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3" gracePeriod=45
Jul 09 17:57:57.285452 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.285396    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:57:57.348623 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348592    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq6d4\" (UniqueName: \"kubernetes.io/projected/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-kube-api-access-tq6d4\") pod \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") "
Jul 09 17:57:57.348778 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348647    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-home\") pod \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") "
Jul 09 17:57:57.348778 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348684    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-model-cache\") pod \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") "
Jul 09 17:57:57.348778 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348767    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tls-certs\") pod \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") "
Jul 09 17:57:57.348933 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348800    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-dshm\") pod \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") "
Jul 09 17:57:57.348933 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348829    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tmp-dir\") pod \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\" (UID: \"5e130fb2-82d4-47b7-aa6f-f835f152b3a6\") "
Jul 09 17:57:57.348933 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348888    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-home" (OuterVolumeSpecName: "home") pod "5e130fb2-82d4-47b7-aa6f-f835f152b3a6" (UID: "5e130fb2-82d4-47b7-aa6f-f835f152b3a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:57:57.349105 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.348976    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e130fb2-82d4-47b7-aa6f-f835f152b3a6" (UID: "5e130fb2-82d4-47b7-aa6f-f835f152b3a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:57:57.349200 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.349180    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e130fb2-82d4-47b7-aa6f-f835f152b3a6" (UID: "5e130fb2-82d4-47b7-aa6f-f835f152b3a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:57:57.349251 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.349201    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:57:57.349251 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.349222    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:57:57.350792 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.350747    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-kube-api-access-tq6d4" (OuterVolumeSpecName: "kube-api-access-tq6d4") pod "5e130fb2-82d4-47b7-aa6f-f835f152b3a6" (UID: "5e130fb2-82d4-47b7-aa6f-f835f152b3a6"). InnerVolumeSpecName "kube-api-access-tq6d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:57:57.350918 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.350826    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-dshm" (OuterVolumeSpecName: "dshm") pod "5e130fb2-82d4-47b7-aa6f-f835f152b3a6" (UID: "5e130fb2-82d4-47b7-aa6f-f835f152b3a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:57:57.350918 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.350835    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e130fb2-82d4-47b7-aa6f-f835f152b3a6" (UID: "5e130fb2-82d4-47b7-aa6f-f835f152b3a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:57:57.449784 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.449754    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:57:57.449784 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.449777    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:57:57.449784 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.449785    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:57:57.449784 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.449794    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq6d4\" (UniqueName: \"kubernetes.io/projected/5e130fb2-82d4-47b7-aa6f-f835f152b3a6-kube-api-access-tq6d4\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:57:57.936599 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.936511    2576 generic.go:356] "Generic (PLEG): container finished" podID="5e130fb2-82d4-47b7-aa6f-f835f152b3a6" containerID="463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3" exitCode=0
Jul 09 17:57:57.936599 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.936550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7" event={"ID":"5e130fb2-82d4-47b7-aa6f-f835f152b3a6","Type":"ContainerDied","Data":"463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3"}
Jul 09 17:57:57.936764 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.936603    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"
Jul 09 17:57:57.936764 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.936631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7" event={"ID":"5e130fb2-82d4-47b7-aa6f-f835f152b3a6","Type":"ContainerDied","Data":"d486fbe5ae2b0b8acac6c585680b360ec11b25f08b08e91130caa26e383e01cf"}
Jul 09 17:57:57.936764 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.936656    2576 scope.go:117] "RemoveContainer" containerID="463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3"
Jul 09 17:57:57.944968 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.944954    2576 scope.go:117] "RemoveContainer" containerID="463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3"
Jul 09 17:57:57.945234 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:57:57.945215    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3\": container with ID starting with 463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3 not found: ID does not exist" containerID="463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3"
Jul 09 17:57:57.945341 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.945239    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3"} err="failed to get container status \"463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3\": rpc error: code = NotFound desc = could not find container \"463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3\": container with ID starting with 463a15ad170f7f3371317d36af492640f90da90e5ddebf377e23470fb2aca1b3 not found: ID does not exist"
Jul 09 17:57:57.953375 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.953348    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"]
Jul 09 17:57:57.956675 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:57.956651    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7"]
Jul 09 17:57:59.670685 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:57:59.670652    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e130fb2-82d4-47b7-aa6f-f835f152b3a6" path="/var/lib/kubelet/pods/5e130fb2-82d4-47b7-aa6f-f835f152b3a6/volumes"
Jul 09 17:58:00.096936 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:58:00.096899    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:58:10.097081 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:58:10.097019    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:58:20.096649 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:58:20.096611    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:58:30.097334 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:58:30.097293    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:58:40.096802 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:58:40.096750    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:58:50.097354 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:58:50.097315    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:59:00.097216 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:00.097125    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:59:10.097159 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:10.097119    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:59:20.106868 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:20.106839    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:59:20.114685 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:20.114659    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:59:21.559780 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:21.559746    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"]
Jul 09 17:59:30.982300 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:30.982269    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:59:30.982810 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:30.982760    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 17:59:36.583734 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:36.583673    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" containerID="cri-o://352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126" gracePeriod=45
Jul 09 17:59:37.108067 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.108014    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 09 17:59:37.342925 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.342867    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:59:37.444030 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.443995    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerID="352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126" exitCode=0
Jul 09 17:59:37.444214 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.444034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" event={"ID":"6ecac386-06ac-4508-b7de-d965e6ff07d0","Type":"ContainerDied","Data":"352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126"}
Jul 09 17:59:37.444214 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.444084    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"
Jul 09 17:59:37.444214 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.444132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq" event={"ID":"6ecac386-06ac-4508-b7de-d965e6ff07d0","Type":"ContainerDied","Data":"14a7df39986e3d46f3a479d877b8eeb9175cbc2ffea16533a0e10572bbff2a00"}
Jul 09 17:59:37.444214 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.444154    2576 scope.go:117] "RemoveContainer" containerID="352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126"
Jul 09 17:59:37.448302 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448280    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-home\") pod \"6ecac386-06ac-4508-b7de-d965e6ff07d0\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") "
Jul 09 17:59:37.448431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448314    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-dshm\") pod \"6ecac386-06ac-4508-b7de-d965e6ff07d0\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") "
Jul 09 17:59:37.448431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecac386-06ac-4508-b7de-d965e6ff07d0-tls-certs\") pod \"6ecac386-06ac-4508-b7de-d965e6ff07d0\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") "
Jul 09 17:59:37.448431 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88fvn\" (UniqueName: \"kubernetes.io/projected/6ecac386-06ac-4508-b7de-d965e6ff07d0-kube-api-access-88fvn\") pod \"6ecac386-06ac-4508-b7de-d965e6ff07d0\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") "
Jul 09 17:59:37.448621 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448561    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-tmp-dir\") pod \"6ecac386-06ac-4508-b7de-d965e6ff07d0\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") "
Jul 09 17:59:37.448677 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448621    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-model-cache\") pod \"6ecac386-06ac-4508-b7de-d965e6ff07d0\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") "
Jul 09 17:59:37.448749 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448725    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-kserve-provision-location\") pod \"6ecac386-06ac-4508-b7de-d965e6ff07d0\" (UID: \"6ecac386-06ac-4508-b7de-d965e6ff07d0\") "
Jul 09 17:59:37.448961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.448935    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-home" (OuterVolumeSpecName: "home") pod "6ecac386-06ac-4508-b7de-d965e6ff07d0" (UID: "6ecac386-06ac-4508-b7de-d965e6ff07d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:59:37.449278 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.449254    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ecac386-06ac-4508-b7de-d965e6ff07d0" (UID: "6ecac386-06ac-4508-b7de-d965e6ff07d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:59:37.451243 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.451210    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecac386-06ac-4508-b7de-d965e6ff07d0-kube-api-access-88fvn" (OuterVolumeSpecName: "kube-api-access-88fvn") pod "6ecac386-06ac-4508-b7de-d965e6ff07d0" (UID: "6ecac386-06ac-4508-b7de-d965e6ff07d0"). InnerVolumeSpecName "kube-api-access-88fvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:59:37.451243 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.451221    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-dshm" (OuterVolumeSpecName: "dshm") pod "6ecac386-06ac-4508-b7de-d965e6ff07d0" (UID: "6ecac386-06ac-4508-b7de-d965e6ff07d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:59:37.451389 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.451264    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecac386-06ac-4508-b7de-d965e6ff07d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ecac386-06ac-4508-b7de-d965e6ff07d0" (UID: "6ecac386-06ac-4508-b7de-d965e6ff07d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:59:37.453922 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.453903    2576 scope.go:117] "RemoveContainer" containerID="1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9"
Jul 09 17:59:37.464160 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.464134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ecac386-06ac-4508-b7de-d965e6ff07d0" (UID: "6ecac386-06ac-4508-b7de-d965e6ff07d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:59:37.469081 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.469058    2576 scope.go:117] "RemoveContainer" containerID="352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126"
Jul 09 17:59:37.469405 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:59:37.469382    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126\": container with ID starting with 352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126 not found: ID does not exist" containerID="352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126"
Jul 09 17:59:37.469468 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.469418    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126"} err="failed to get container status \"352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126\": rpc error: code = NotFound desc = could not find container \"352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126\": container with ID starting with 352f7df400ff5d8f9cea54a6b2b0465a5354b5a4cfba6b798d38598a3c09d126 not found: ID does not exist"
Jul 09 17:59:37.469468 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.469441    2576 scope.go:117] "RemoveContainer" containerID="1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9"
Jul 09 17:59:37.469712 ip-10-0-132-16 kubenswrapper[2576]: E0709 17:59:37.469681    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9\": container with ID starting with 1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9 not found: ID does not exist" containerID="1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9"
Jul 09 17:59:37.469770 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.469713    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9"} err="failed to get container status \"1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9\": rpc error: code = NotFound desc = could not find container \"1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9\": container with ID starting with 1136f0ef997f09d1b5ed0ea8d4b8b521b7ea2e7c41f142ad0a0783c8cf5b10d9 not found: ID does not exist"
Jul 09 17:59:37.513404 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.513361    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ecac386-06ac-4508-b7de-d965e6ff07d0" (UID: "6ecac386-06ac-4508-b7de-d965e6ff07d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:59:37.550401 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.550365    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:59:37.550401 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.550396    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:59:37.550401 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.550407    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:59:37.550615 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.550414    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecac386-06ac-4508-b7de-d965e6ff07d0-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:59:37.550615 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.550424    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88fvn\" (UniqueName: \"kubernetes.io/projected/6ecac386-06ac-4508-b7de-d965e6ff07d0-kube-api-access-88fvn\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:59:37.550615 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.550431    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:59:37.550615 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.550439    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecac386-06ac-4508-b7de-d965e6ff07d0-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 17:59:37.761195 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.761167    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"]
Jul 09 17:59:37.764719 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:37.764695    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq"]
Jul 09 17:59:39.668961 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:39.668932    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" path="/var/lib/kubelet/pods/6ecac386-06ac-4508-b7de-d965e6ff07d0/volumes"
Jul 09 17:59:52.383320 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383281    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"]
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383650    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main"
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383663    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main"
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383675    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="storage-initializer"
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383681    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="storage-initializer"
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383690    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e130fb2-82d4-47b7-aa6f-f835f152b3a6" containerName="main"
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383695    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e130fb2-82d4-47b7-aa6f-f835f152b3a6" containerName="main"
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383756    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecac386-06ac-4508-b7de-d965e6ff07d0" containerName="main"
Jul 09 17:59:52.383916 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.383763    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e130fb2-82d4-47b7-aa6f-f835f152b3a6" containerName="main"
Jul 09 17:59:52.388676 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.388660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.391969 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.391942    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 09 17:59:52.392135 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.392076    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 17:59:52.392217 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.391988    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:59:52.392315 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.392291    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fg56t\""
Jul 09 17:59:52.392444 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.392375    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 17:59:52.454785 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.454754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"]
Jul 09 17:59:52.487022 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.486981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zdf\" (UniqueName: \"kubernetes.io/projected/c558862a-82bd-4184-ab30-8c0b2b128cbb-kube-api-access-s2zdf\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.487022 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.487022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-dshm\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.487266 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.487123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-tmp-dir\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.487266 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.487172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-model-cache\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.487266 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.487208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-kserve-provision-location\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.487266 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.487237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-home\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.487412 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.487301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558862a-82bd-4184-ab30-8c0b2b128cbb-tls-certs\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.588689 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.588650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-home\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.588878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.588708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558862a-82bd-4184-ab30-8c0b2b128cbb-tls-certs\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.588878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.588743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zdf\" (UniqueName: \"kubernetes.io/projected/c558862a-82bd-4184-ab30-8c0b2b128cbb-kube-api-access-s2zdf\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.588878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.588761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-dshm\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.588878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.588798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-tmp-dir\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.588878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.588830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-model-cache\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.588878 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.588856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-kserve-provision-location\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.589232 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.589171    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-home\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.589286 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.589232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-tmp-dir\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.589393 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.589372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-kserve-provision-location\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.589432 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.589391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-model-cache\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.591080 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.591059    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-dshm\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.591338 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.591317    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558862a-82bd-4184-ab30-8c0b2b128cbb-tls-certs\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.597256 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.597230    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zdf\" (UniqueName: \"kubernetes.io/projected/c558862a-82bd-4184-ab30-8c0b2b128cbb-kube-api-access-s2zdf\") pod \"auth-disabled-test-kserve-5ff8d867d8-w25vd\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.700451 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.700349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 17:59:52.855174 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:52.855138    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"]
Jul 09 17:59:52.858378 ip-10-0-132-16 kubenswrapper[2576]: W0709 17:59:52.858341    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc558862a_82bd_4184_ab30_8c0b2b128cbb.slice/crio-6f99acbb8ba79113445757d855ef1fb70a4ad0147cdae91139b1640c28b682ad WatchSource:0}: Error finding container 6f99acbb8ba79113445757d855ef1fb70a4ad0147cdae91139b1640c28b682ad: Status 404 returned error can't find the container with id 6f99acbb8ba79113445757d855ef1fb70a4ad0147cdae91139b1640c28b682ad
Jul 09 17:59:53.525656 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:53.525622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" event={"ID":"c558862a-82bd-4184-ab30-8c0b2b128cbb","Type":"ContainerStarted","Data":"392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a"}
Jul 09 17:59:53.526023 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:53.525681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" event={"ID":"c558862a-82bd-4184-ab30-8c0b2b128cbb","Type":"ContainerStarted","Data":"6f99acbb8ba79113445757d855ef1fb70a4ad0147cdae91139b1640c28b682ad"}
Jul 09 17:59:56.543736 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:56.543650    2576 generic.go:356] "Generic (PLEG): container finished" podID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerID="392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a" exitCode=0
Jul 09 17:59:56.543736 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:56.543710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" event={"ID":"c558862a-82bd-4184-ab30-8c0b2b128cbb","Type":"ContainerDied","Data":"392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a"}
Jul 09 17:59:57.553718 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:57.553680    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" event={"ID":"c558862a-82bd-4184-ab30-8c0b2b128cbb","Type":"ContainerStarted","Data":"cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957"}
Jul 09 17:59:57.584233 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:57.584179    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podStartSLOduration=2.310634307 podStartE2EDuration="5.584163209s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.273528902s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:59:57.581240929 +0000 UTC m=+1227.112017983" watchObservedRunningTime="2026-07-09 17:59:57.584163209 +0000 UTC m=+1227.114940238"
Jul 09 17:59:57.700840 ip-10-0-132-16 kubenswrapper[2576]: I0709 17:59:57.700791    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 18:00:02.701410 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:02.701361    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 18:00:02.702721 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:02.702693    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:00:07.611421 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.611386    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"]
Jul 09 18:00:07.632466 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.632439    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"]
Jul 09 18:00:07.632619 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.632536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.637337 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.637311    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 09 18:00:07.637479 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.637403    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-6l6v2\""
Jul 09 18:00:07.735133 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.735014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.735295 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.735179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.735295 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.735217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgmr\" (UniqueName: \"kubernetes.io/projected/915d0b9c-2826-46c3-8dd9-057fe88c5d89-kube-api-access-ftgmr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.836756 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.836710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.837678 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.837651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.837888 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.837866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgmr\" (UniqueName: \"kubernetes.io/projected/915d0b9c-2826-46c3-8dd9-057fe88c5d89-kube-api-access-ftgmr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.838194 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.838154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.849517 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.849491    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.852353 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.852327    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgmr\" (UniqueName: \"kubernetes.io/projected/915d0b9c-2826-46c3-8dd9-057fe88c5d89-kube-api-access-ftgmr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:07.942367 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:07.942293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:08.101654 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:08.101629    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"]
Jul 09 18:00:08.104126 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:00:08.104089    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915d0b9c_2826_46c3_8dd9_057fe88c5d89.slice/crio-58871ac1489c1f4c19f5df3f6d32656b5efe574cb341b556950877cf7386d29e WatchSource:0}: Error finding container 58871ac1489c1f4c19f5df3f6d32656b5efe574cb341b556950877cf7386d29e: Status 404 returned error can't find the container with id 58871ac1489c1f4c19f5df3f6d32656b5efe574cb341b556950877cf7386d29e
Jul 09 18:00:08.106162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:08.106146    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:00:08.610426 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:08.610392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" event={"ID":"915d0b9c-2826-46c3-8dd9-057fe88c5d89","Type":"ContainerStarted","Data":"58871ac1489c1f4c19f5df3f6d32656b5efe574cb341b556950877cf7386d29e"}
Jul 09 18:00:12.701220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:12.701173    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:00:19.670471 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:19.670435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" event={"ID":"915d0b9c-2826-46c3-8dd9-057fe88c5d89","Type":"ContainerStarted","Data":"e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3"}
Jul 09 18:00:19.670471 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:19.670492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:19.693845 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:19.693777    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" podStartSLOduration=1.5404947770000001 podStartE2EDuration="12.693754111s" totalImagesPullingTime="11.153259334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:00:19.68991726 +0000 UTC m=+1249.220694315" watchObservedRunningTime="2026-07-09 18:00:19.693754111 +0000 UTC m=+1249.224531143"
Jul 09 18:00:22.701905 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:22.701804    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:00:32.701285 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:32.701233    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:00:42.700933 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:42.700890    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:00:50.677637 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:50.677607    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:51.714322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:51.714290    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"]
Jul 09 18:00:51.725327 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:00:51.725283    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:00:51.725327 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:00:51.725327 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:00:51.725327 ip-10-0-132-16 kubenswrapper[2576]:  > containerID="e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3" cmd=["/bin/sleep","15"]
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:51.725341    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" message="" err=<
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]:  >
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:00:51.725354    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" podUID="915d0b9c-2826-46c3-8dd9-057fe88c5d89" containerName="main" containerID="cri-o://e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3"
Jul 09 18:00:51.725573 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:51.725406    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" podUID="915d0b9c-2826-46c3-8dd9-057fe88c5d89" containerName="main" containerID="cri-o://e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3" gracePeriod=60
Jul 09 18:00:51.983590 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:51.983563    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:52.049755 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.049720    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tls-certs\") pod \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") "
Jul 09 18:00:52.049934 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.049760    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftgmr\" (UniqueName: \"kubernetes.io/projected/915d0b9c-2826-46c3-8dd9-057fe88c5d89-kube-api-access-ftgmr\") pod \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") "
Jul 09 18:00:52.049934 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.049860    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tokenizer-uds\") pod \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\" (UID: \"915d0b9c-2826-46c3-8dd9-057fe88c5d89\") "
Jul 09 18:00:52.050145 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.050119    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "915d0b9c-2826-46c3-8dd9-057fe88c5d89" (UID: "915d0b9c-2826-46c3-8dd9-057fe88c5d89"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:00:52.051879 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.051851    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "915d0b9c-2826-46c3-8dd9-057fe88c5d89" (UID: "915d0b9c-2826-46c3-8dd9-057fe88c5d89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:00:52.051971 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.051884    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915d0b9c-2826-46c3-8dd9-057fe88c5d89-kube-api-access-ftgmr" (OuterVolumeSpecName: "kube-api-access-ftgmr") pod "915d0b9c-2826-46c3-8dd9-057fe88c5d89" (UID: "915d0b9c-2826-46c3-8dd9-057fe88c5d89"). InnerVolumeSpecName "kube-api-access-ftgmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:00:52.151282 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.151246    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:00:52.151282 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.151276    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgmr\" (UniqueName: \"kubernetes.io/projected/915d0b9c-2826-46c3-8dd9-057fe88c5d89-kube-api-access-ftgmr\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:00:52.151282 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.151286    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/915d0b9c-2826-46c3-8dd9-057fe88c5d89-tokenizer-uds\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:00:52.700767 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.700720    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:00:52.838466 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.838424    2576 generic.go:356] "Generic (PLEG): container finished" podID="915d0b9c-2826-46c3-8dd9-057fe88c5d89" containerID="e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3" exitCode=0
Jul 09 18:00:52.838466 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.838464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"
Jul 09 18:00:52.838995 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.838504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" event={"ID":"915d0b9c-2826-46c3-8dd9-057fe88c5d89","Type":"ContainerDied","Data":"e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3"}
Jul 09 18:00:52.838995 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.838562    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs" event={"ID":"915d0b9c-2826-46c3-8dd9-057fe88c5d89","Type":"ContainerDied","Data":"58871ac1489c1f4c19f5df3f6d32656b5efe574cb341b556950877cf7386d29e"}
Jul 09 18:00:52.838995 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.838591    2576 scope.go:117] "RemoveContainer" containerID="e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3"
Jul 09 18:00:52.848475 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.848448    2576 scope.go:117] "RemoveContainer" containerID="e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3"
Jul 09 18:00:52.848771 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:00:52.848754    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3\": container with ID starting with e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3 not found: ID does not exist" containerID="e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3"
Jul 09 18:00:52.848824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.848779    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3"} err="failed to get container status \"e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3\": rpc error: code = NotFound desc = could not find container \"e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3\": container with ID starting with e2996b0a67783b91db1bfeb601272b8ae819e7d0c195953b43441f3a30e8b4f3 not found: ID does not exist"
Jul 09 18:00:52.860901 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.860875    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"]
Jul 09 18:00:52.865137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:52.865113    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs"]
Jul 09 18:00:53.669726 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:00:53.669697    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915d0b9c-2826-46c3-8dd9-057fe88c5d89" path="/var/lib/kubelet/pods/915d0b9c-2826-46c3-8dd9-057fe88c5d89/volumes"
Jul 09 18:01:02.700971 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:02.700923    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:01:12.700862 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:12.700823    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:01:22.700822 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:22.700783    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:01:32.701177 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:32.701132    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:01:35.380406 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.380377    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"]
Jul 09 18:01:35.380884 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.380784    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915d0b9c-2826-46c3-8dd9-057fe88c5d89" containerName="main"
Jul 09 18:01:35.380884 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.380796    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="915d0b9c-2826-46c3-8dd9-057fe88c5d89" containerName="main"
Jul 09 18:01:35.380884 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.380864    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="915d0b9c-2826-46c3-8dd9-057fe88c5d89" containerName="main"
Jul 09 18:01:35.383330 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.383311    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.386065 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.386026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 09 18:01:35.386162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.386031    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:01:35.399230 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.399206    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"]
Jul 09 18:01:35.439355 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.439506 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439379    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.439506 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.439506 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.439742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.439742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.439742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.439742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.439724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjsp\" (UniqueName: \"kubernetes.io/projected/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kube-api-access-5wjsp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540389 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjsp\" (UniqueName: \"kubernetes.io/projected/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kube-api-access-5wjsp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.540826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.541028 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.540946    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.541119 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.541090    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.541464 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.541442    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.542952 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.542921    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.543162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.543146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.551385 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.551361    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjsp\" (UniqueName: \"kubernetes.io/projected/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kube-api-access-5wjsp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.701177 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.701077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:35.855859 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:35.855822    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"]
Jul 09 18:01:35.858790 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:01:35.858760    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94ab9d1_d00e_4e90_b9ad_10e3e357781e.slice/crio-19bf190f8eccbae678186fcff14c1f6dfe543f0610b46b7790425bad4312d22d WatchSource:0}: Error finding container 19bf190f8eccbae678186fcff14c1f6dfe543f0610b46b7790425bad4312d22d: Status 404 returned error can't find the container with id 19bf190f8eccbae678186fcff14c1f6dfe543f0610b46b7790425bad4312d22d
Jul 09 18:01:36.055899 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:36.055820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" event={"ID":"f94ab9d1-d00e-4e90-b9ad-10e3e357781e","Type":"ContainerStarted","Data":"af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a"}
Jul 09 18:01:36.055899 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:36.055870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" event={"ID":"f94ab9d1-d00e-4e90-b9ad-10e3e357781e","Type":"ContainerStarted","Data":"19bf190f8eccbae678186fcff14c1f6dfe543f0610b46b7790425bad4312d22d"}
Jul 09 18:01:40.081397 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:40.081369    2576 generic.go:356] "Generic (PLEG): container finished" podID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerID="af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a" exitCode=0
Jul 09 18:01:40.081880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:40.081437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" event={"ID":"f94ab9d1-d00e-4e90-b9ad-10e3e357781e","Type":"ContainerDied","Data":"af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a"}
Jul 09 18:01:41.088368 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:41.088327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" event={"ID":"f94ab9d1-d00e-4e90-b9ad-10e3e357781e","Type":"ContainerStarted","Data":"3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6"}
Jul 09 18:01:41.120943 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:41.120895    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podStartSLOduration=2.258193144 podStartE2EDuration="6.120880966s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.862687822s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:01:41.118734922 +0000 UTC m=+1330.649511975" watchObservedRunningTime="2026-07-09 18:01:41.120880966 +0000 UTC m=+1330.651657995"
Jul 09 18:01:41.701337 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:41.701300    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:42.701515 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:42.701472    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:01:45.701903 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:45.701858    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:01:45.703666 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:45.703628    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:01:52.711653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:52.711574    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 18:01:52.719355 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:52.719332    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 18:01:55.701700 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:55.701658    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:01:57.089822 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:01:57.089784    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"]
Jul 09 18:02:05.702008 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:05.701964    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:02:12.113640 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.113592    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" containerID="cri-o://cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957" gracePeriod=45
Jul 09 18:02:12.712490 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.712451    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 09 18:02:12.852713 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.852686    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 18:02:12.903918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.903852    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-tmp-dir\") pod \"c558862a-82bd-4184-ab30-8c0b2b128cbb\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") "
Jul 09 18:02:12.903918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.903895    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2zdf\" (UniqueName: \"kubernetes.io/projected/c558862a-82bd-4184-ab30-8c0b2b128cbb-kube-api-access-s2zdf\") pod \"c558862a-82bd-4184-ab30-8c0b2b128cbb\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") "
Jul 09 18:02:12.904121 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.903931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-dshm\") pod \"c558862a-82bd-4184-ab30-8c0b2b128cbb\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") "
Jul 09 18:02:12.904121 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.903964    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-kserve-provision-location\") pod \"c558862a-82bd-4184-ab30-8c0b2b128cbb\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") "
Jul 09 18:02:12.904121 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.904009    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558862a-82bd-4184-ab30-8c0b2b128cbb-tls-certs\") pod \"c558862a-82bd-4184-ab30-8c0b2b128cbb\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") "
Jul 09 18:02:12.904285 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.904170    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-home\") pod \"c558862a-82bd-4184-ab30-8c0b2b128cbb\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") "
Jul 09 18:02:12.904285 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.904254    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-model-cache\") pod \"c558862a-82bd-4184-ab30-8c0b2b128cbb\" (UID: \"c558862a-82bd-4184-ab30-8c0b2b128cbb\") "
Jul 09 18:02:12.904708 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.904674    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c558862a-82bd-4184-ab30-8c0b2b128cbb" (UID: "c558862a-82bd-4184-ab30-8c0b2b128cbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:02:12.905189 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.905157    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-home" (OuterVolumeSpecName: "home") pod "c558862a-82bd-4184-ab30-8c0b2b128cbb" (UID: "c558862a-82bd-4184-ab30-8c0b2b128cbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:02:12.906554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.906516    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-dshm" (OuterVolumeSpecName: "dshm") pod "c558862a-82bd-4184-ab30-8c0b2b128cbb" (UID: "c558862a-82bd-4184-ab30-8c0b2b128cbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:02:12.906554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.906536    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c558862a-82bd-4184-ab30-8c0b2b128cbb-kube-api-access-s2zdf" (OuterVolumeSpecName: "kube-api-access-s2zdf") pod "c558862a-82bd-4184-ab30-8c0b2b128cbb" (UID: "c558862a-82bd-4184-ab30-8c0b2b128cbb"). InnerVolumeSpecName "kube-api-access-s2zdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:02:12.906695 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.906611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c558862a-82bd-4184-ab30-8c0b2b128cbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c558862a-82bd-4184-ab30-8c0b2b128cbb" (UID: "c558862a-82bd-4184-ab30-8c0b2b128cbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:02:12.916100 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.916076    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c558862a-82bd-4184-ab30-8c0b2b128cbb" (UID: "c558862a-82bd-4184-ab30-8c0b2b128cbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:02:12.980453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:12.980424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c558862a-82bd-4184-ab30-8c0b2b128cbb" (UID: "c558862a-82bd-4184-ab30-8c0b2b128cbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:02:13.005812 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.005789    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:02:13.005812 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.005809    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:02:13.005939 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.005820    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2zdf\" (UniqueName: \"kubernetes.io/projected/c558862a-82bd-4184-ab30-8c0b2b128cbb-kube-api-access-s2zdf\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:02:13.005939 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.005828    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:02:13.005939 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.005837    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:02:13.005939 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.005848    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558862a-82bd-4184-ab30-8c0b2b128cbb-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:02:13.005939 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.005856    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c558862a-82bd-4184-ab30-8c0b2b128cbb-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:02:13.250883 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.250804    2576 generic.go:356] "Generic (PLEG): container finished" podID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerID="cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957" exitCode=0
Jul 09 18:02:13.250883 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.250854    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"
Jul 09 18:02:13.251342 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.250898    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" event={"ID":"c558862a-82bd-4184-ab30-8c0b2b128cbb","Type":"ContainerDied","Data":"cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957"}
Jul 09 18:02:13.251342 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.250969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd" event={"ID":"c558862a-82bd-4184-ab30-8c0b2b128cbb","Type":"ContainerDied","Data":"6f99acbb8ba79113445757d855ef1fb70a4ad0147cdae91139b1640c28b682ad"}
Jul 09 18:02:13.251342 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.251002    2576 scope.go:117] "RemoveContainer" containerID="cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957"
Jul 09 18:02:13.261404 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.261383    2576 scope.go:117] "RemoveContainer" containerID="392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a"
Jul 09 18:02:13.269374 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.269360    2576 scope.go:117] "RemoveContainer" containerID="cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957"
Jul 09 18:02:13.269658 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:02:13.269642    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957\": container with ID starting with cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957 not found: ID does not exist" containerID="cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957"
Jul 09 18:02:13.269740 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.269668    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957"} err="failed to get container status \"cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957\": rpc error: code = NotFound desc = could not find container \"cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957\": container with ID starting with cc597ac42d10dcd72782ff144f98a93a38c32a33cee377e469ba6af0e250e957 not found: ID does not exist"
Jul 09 18:02:13.269740 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.269689    2576 scope.go:117] "RemoveContainer" containerID="392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a"
Jul 09 18:02:13.269901 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:02:13.269885    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a\": container with ID starting with 392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a not found: ID does not exist" containerID="392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a"
Jul 09 18:02:13.269951 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.269909    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a"} err="failed to get container status \"392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a\": rpc error: code = NotFound desc = could not find container \"392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a\": container with ID starting with 392b7b1a4aafdf9e78795326e0606c92c0492f0efa7a5149614021ba30e2de8a not found: ID does not exist"
Jul 09 18:02:13.274977 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.274954    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"]
Jul 09 18:02:13.278240 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.278220    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd"]
Jul 09 18:02:13.670548 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:13.670521    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" path="/var/lib/kubelet/pods/c558862a-82bd-4184-ab30-8c0b2b128cbb/volumes"
Jul 09 18:02:15.702283 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:15.702245    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:02:25.701841 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:25.701803    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:02:35.701569 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:35.701527    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:02:45.701943 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:45.701892    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:02:55.712116 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:55.712088    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:02:55.720210 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:02:55.720185    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:03:16.911248 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.911217    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"]
Jul 09 18:03:16.911723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.911584    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="storage-initializer"
Jul 09 18:03:16.911723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.911595    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="storage-initializer"
Jul 09 18:03:16.911723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.911610    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main"
Jul 09 18:03:16.911723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.911615    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main"
Jul 09 18:03:16.911723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.911666    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c558862a-82bd-4184-ab30-8c0b2b128cbb" containerName="main"
Jul 09 18:03:16.914417 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.914400    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:16.917248 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.917224    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 09 18:03:16.926433 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.926401    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"]
Jul 09 18:03:16.993746 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.993709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmznk\" (UniqueName: \"kubernetes.io/projected/abd2015c-3d81-47e3-8843-65c456a014e4-kube-api-access-fmznk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:16.993918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.993777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:16.993918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.993879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd2015c-3d81-47e3-8843-65c456a014e4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:16.994009 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.993925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:16.994079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.994034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:16.994133 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.994077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:16.994133 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:16.994099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095106 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd2015c-3d81-47e3-8843-65c456a014e4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095560 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmznk\" (UniqueName: \"kubernetes.io/projected/abd2015c-3d81-47e3-8843-65c456a014e4-kube-api-access-fmznk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095560 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095654 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095797 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.095912 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.095800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.097805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.097786    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd2015c-3d81-47e3-8843-65c456a014e4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.097877 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.097789    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.104468 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.104443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmznk\" (UniqueName: \"kubernetes.io/projected/abd2015c-3d81-47e3-8843-65c456a014e4-kube-api-access-fmznk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.225364 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.225289    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:17.412796 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.412764    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"]
Jul 09 18:03:17.416336 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:03:17.416304    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd2015c_3d81_47e3_8843_65c456a014e4.slice/crio-5d4991f718ec818de8e1e0c041db038731dd920fb6c6729816ebbbab6edd9c80 WatchSource:0}: Error finding container 5d4991f718ec818de8e1e0c041db038731dd920fb6c6729816ebbbab6edd9c80: Status 404 returned error can't find the container with id 5d4991f718ec818de8e1e0c041db038731dd920fb6c6729816ebbbab6edd9c80
Jul 09 18:03:17.573134 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.573098    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" event={"ID":"abd2015c-3d81-47e3-8843-65c456a014e4","Type":"ContainerStarted","Data":"6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92"}
Jul 09 18:03:17.573309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:17.573156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" event={"ID":"abd2015c-3d81-47e3-8843-65c456a014e4","Type":"ContainerStarted","Data":"5d4991f718ec818de8e1e0c041db038731dd920fb6c6729816ebbbab6edd9c80"}
Jul 09 18:03:19.620911 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:19.620876    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"]
Jul 09 18:03:21.609707 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:21.609678    2576 generic.go:356] "Generic (PLEG): container finished" podID="abd2015c-3d81-47e3-8843-65c456a014e4" containerID="6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92" exitCode=0
Jul 09 18:03:21.610069 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:21.609748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" event={"ID":"abd2015c-3d81-47e3-8843-65c456a014e4","Type":"ContainerDied","Data":"6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92"}
Jul 09 18:03:22.617738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:22.617704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" event={"ID":"abd2015c-3d81-47e3-8843-65c456a014e4","Type":"ContainerStarted","Data":"63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2"}
Jul 09 18:03:22.642793 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:22.642739    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podStartSLOduration=2.8432539329999997 podStartE2EDuration="6.642723124s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.799469191s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:22.639816322 +0000 UTC m=+1432.170593353" watchObservedRunningTime="2026-07-09 18:03:22.642723124 +0000 UTC m=+1432.173500153"
Jul 09 18:03:23.226329 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:23.226293    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:27.226276 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:27.226233    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:03:27.227723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:27.227696    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:03:34.653890 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:34.653846    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" containerID="cri-o://3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6" gracePeriod=45
Jul 09 18:03:34.713098 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:34.713051    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 09 18:03:35.302694 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.302665    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:03:35.372834 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.372765    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tls-certs\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.372834 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.372808    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-home\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.372834 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.372834    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tmp-dir\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.373111 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.372856    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kserve-provision-location\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.373111 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.372886    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-cabundle-cert\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.373111 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.372918    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-dshm\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.373111 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.372943    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wjsp\" (UniqueName: \"kubernetes.io/projected/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kube-api-access-5wjsp\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.373309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.373127    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-model-cache\") pod \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\" (UID: \"f94ab9d1-d00e-4e90-b9ad-10e3e357781e\") "
Jul 09 18:03:35.373309 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.373222    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:03:35.373625 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.373456    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-cabundle-cert\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.373827 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.373727    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:35.373827 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.373661    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-home" (OuterVolumeSpecName: "home") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:35.375480 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.375442    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kube-api-access-5wjsp" (OuterVolumeSpecName: "kube-api-access-5wjsp") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "kube-api-access-5wjsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:03:35.375751 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.375726    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-dshm" (OuterVolumeSpecName: "dshm") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:35.375844 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.375754    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:03:35.386017 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.385993    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:35.437647 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.437608    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f94ab9d1-d00e-4e90-b9ad-10e3e357781e" (UID: "f94ab9d1-d00e-4e90-b9ad-10e3e357781e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:35.474898 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.474871    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.474898 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.474894    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.474898 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.474902    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.475082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.474911    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.475082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.474920    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.475082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.474929    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjsp\" (UniqueName: \"kubernetes.io/projected/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-kube-api-access-5wjsp\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.475082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.474939    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94ab9d1-d00e-4e90-b9ad-10e3e357781e-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:03:35.684067 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.684026    2576 generic.go:356] "Generic (PLEG): container finished" podID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerID="3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6" exitCode=0
Jul 09 18:03:35.684505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.684099    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"
Jul 09 18:03:35.684620 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.684590    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" event={"ID":"f94ab9d1-d00e-4e90-b9ad-10e3e357781e","Type":"ContainerDied","Data":"3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6"}
Jul 09 18:03:35.684714 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.684656    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl" event={"ID":"f94ab9d1-d00e-4e90-b9ad-10e3e357781e","Type":"ContainerDied","Data":"19bf190f8eccbae678186fcff14c1f6dfe543f0610b46b7790425bad4312d22d"}
Jul 09 18:03:35.684714 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.684695    2576 scope.go:117] "RemoveContainer" containerID="3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6"
Jul 09 18:03:35.698143 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.698131    2576 scope.go:117] "RemoveContainer" containerID="af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a"
Jul 09 18:03:35.698547 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:03:35.698521    2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94ab9d1_d00e_4e90_b9ad_10e3e357781e.slice\": RecentStats: unable to find data in memory cache]"
Jul 09 18:03:35.705751 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.705690    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"]
Jul 09 18:03:35.709879 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.709423    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl"]
Jul 09 18:03:35.714666 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.714645    2576 scope.go:117] "RemoveContainer" containerID="3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6"
Jul 09 18:03:35.715133 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:03:35.715109    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6\": container with ID starting with 3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6 not found: ID does not exist" containerID="3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6"
Jul 09 18:03:35.715219 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.715141    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6"} err="failed to get container status \"3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6\": rpc error: code = NotFound desc = could not find container \"3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6\": container with ID starting with 3625eb3b0248b15d0ed2659bcaf6d37d2e6277a6bd21b8e1f59932d9f1225df6 not found: ID does not exist"
Jul 09 18:03:35.715219 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.715159    2576 scope.go:117] "RemoveContainer" containerID="af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a"
Jul 09 18:03:35.715483 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:03:35.715460    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a\": container with ID starting with af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a not found: ID does not exist" containerID="af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a"
Jul 09 18:03:35.715545 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:35.715494    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a"} err="failed to get container status \"af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a\": rpc error: code = NotFound desc = could not find container \"af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a\": container with ID starting with af2e6302838d93a97025e3d25bcbb8be32d6279a0c09e8e8fa4f28737749066a not found: ID does not exist"
Jul 09 18:03:37.225854 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:37.225808    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:03:37.669766 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:37.669738    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" path="/var/lib/kubelet/pods/f94ab9d1-d00e-4e90-b9ad-10e3e357781e/volumes"
Jul 09 18:03:43.902442 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.902401    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"]
Jul 09 18:03:43.902922 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.902788    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main"
Jul 09 18:03:43.902922 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.902802    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main"
Jul 09 18:03:43.902922 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.902828    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="storage-initializer"
Jul 09 18:03:43.902922 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.902834    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="storage-initializer"
Jul 09 18:03:43.902922 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.902898    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94ab9d1-d00e-4e90-b9ad-10e3e357781e" containerName="main"
Jul 09 18:03:43.906167 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.906147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:43.908952 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.908905    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:03:43.909093 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.909018    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 09 18:03:43.919640 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:43.919617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"]
Jul 09 18:03:44.051440 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jz2\" (UniqueName: \"kubernetes.io/projected/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-kube-api-access-76jz2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.051440 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.051689 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.051689 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.051689 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.051689 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.051689 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.051886 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.051709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152501 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jz2\" (UniqueName: \"kubernetes.io/projected/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-kube-api-access-76jz2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152501 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152501 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152771 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152771 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152771 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152771 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.152771 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.152688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.153157 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.153133    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.153298 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.153174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.153414 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.153319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.153536 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.153356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.153536 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.153524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.155956 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.155931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.156083 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.156069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.166011 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.165981    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jz2\" (UniqueName: \"kubernetes.io/projected/ce9d7d7f-5c2b-4845-a794-a578803d5d9a-kube-api-access-76jz2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv\" (UID: \"ce9d7d7f-5c2b-4845-a794-a578803d5d9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.216606 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.216582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:44.358460 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.358437    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"]
Jul 09 18:03:44.360226 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:03:44.360199    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9d7d7f_5c2b_4845_a794_a578803d5d9a.slice/crio-f17ea418a5af49931523086446feff2761e7f6d75cda51421f409d00794c85e5 WatchSource:0}: Error finding container f17ea418a5af49931523086446feff2761e7f6d75cda51421f409d00794c85e5: Status 404 returned error can't find the container with id f17ea418a5af49931523086446feff2761e7f6d75cda51421f409d00794c85e5
Jul 09 18:03:44.727587 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.727498    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" event={"ID":"ce9d7d7f-5c2b-4845-a794-a578803d5d9a","Type":"ContainerStarted","Data":"96e21b2772a740494c6a78d3731965dd442800d285dfa2743489fec1f489d206"}
Jul 09 18:03:44.727587 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:44.727570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" event={"ID":"ce9d7d7f-5c2b-4845-a794-a578803d5d9a","Type":"ContainerStarted","Data":"f17ea418a5af49931523086446feff2761e7f6d75cda51421f409d00794c85e5"}
Jul 09 18:03:47.226723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:47.226659    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:03:48.750274 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:48.750237    2576 generic.go:356] "Generic (PLEG): container finished" podID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerID="96e21b2772a740494c6a78d3731965dd442800d285dfa2743489fec1f489d206" exitCode=0
Jul 09 18:03:48.750620 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:48.750316    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" event={"ID":"ce9d7d7f-5c2b-4845-a794-a578803d5d9a","Type":"ContainerDied","Data":"96e21b2772a740494c6a78d3731965dd442800d285dfa2743489fec1f489d206"}
Jul 09 18:03:49.757844 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:49.757807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" event={"ID":"ce9d7d7f-5c2b-4845-a794-a578803d5d9a","Type":"ContainerStarted","Data":"3ede921af921e7f35aa4d5153110238108ca9cba627a25c66cfdb726893bae27"}
Jul 09 18:03:49.782322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:49.782279    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podStartSLOduration=3.115643851 podStartE2EDuration="6.782263648s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.666619797s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:49.779769197 +0000 UTC m=+1459.310546250" watchObservedRunningTime="2026-07-09 18:03:49.782263648 +0000 UTC m=+1459.313040675"
Jul 09 18:03:50.216837 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:50.216804    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:54.217058 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:54.217007    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:03:54.218399 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:54.218367    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podUID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 09 18:03:57.226306 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:03:57.226256    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:04:04.217675 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:04.217619    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podUID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 09 18:04:07.226670 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:07.226619    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:04:14.217222 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:14.217179    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podUID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 09 18:04:17.226292 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:17.226239    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:04:24.217223 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:24.217184    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podUID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 09 18:04:27.226325 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:27.226282    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:04:31.018010 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:31.017982    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:04:31.019900 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:31.019878    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:04:34.217284 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:34.217237    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podUID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 09 18:04:37.226847 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:37.226800    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:04:44.218162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:44.218122    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podUID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 09 18:04:47.226646 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:47.226600    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:04:54.217893 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:54.217798    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv" podUID="ce9d7d7f-5c2b-4845-a794-a578803d5d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 09 18:04:57.225919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:04:57.225866    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:05:04.226874 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:04.226845    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:05:04.234547 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:04.234522    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv"
Jul 09 18:05:07.226421 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:07.226370    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:05:17.236940 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:17.236911    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:05:17.244575 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:17.244551    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:05:18.347729 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:18.347698    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:05:18.357848 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:18.357829    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:05:22.007350 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:22.007311    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"]
Jul 09 18:05:37.032152 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.032082    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" containerID="cri-o://63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2" gracePeriod=45
Jul 09 18:05:37.237808 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.237767    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 09 18:05:37.773590 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.773568    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:05:37.837288 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-dshm\") pod \"abd2015c-3d81-47e3-8843-65c456a014e4\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") "
Jul 09 18:05:37.837431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837295    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-tmp-dir\") pod \"abd2015c-3d81-47e3-8843-65c456a014e4\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") "
Jul 09 18:05:37.837431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837317    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-model-cache\") pod \"abd2015c-3d81-47e3-8843-65c456a014e4\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") "
Jul 09 18:05:37.837431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837332    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-kserve-provision-location\") pod \"abd2015c-3d81-47e3-8843-65c456a014e4\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") "
Jul 09 18:05:37.837431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837356    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmznk\" (UniqueName: \"kubernetes.io/projected/abd2015c-3d81-47e3-8843-65c456a014e4-kube-api-access-fmznk\") pod \"abd2015c-3d81-47e3-8843-65c456a014e4\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") "
Jul 09 18:05:37.837431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837384    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd2015c-3d81-47e3-8843-65c456a014e4-tls-certs\") pod \"abd2015c-3d81-47e3-8843-65c456a014e4\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") "
Jul 09 18:05:37.837702 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837552    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "abd2015c-3d81-47e3-8843-65c456a014e4" (UID: "abd2015c-3d81-47e3-8843-65c456a014e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:37.837702 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837591    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-home\") pod \"abd2015c-3d81-47e3-8843-65c456a014e4\" (UID: \"abd2015c-3d81-47e3-8843-65c456a014e4\") "
Jul 09 18:05:37.838018 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.837978    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:05:37.838123 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.838107    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-home" (OuterVolumeSpecName: "home") pod "abd2015c-3d81-47e3-8843-65c456a014e4" (UID: "abd2015c-3d81-47e3-8843-65c456a014e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:37.839355 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.839332    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-dshm" (OuterVolumeSpecName: "dshm") pod "abd2015c-3d81-47e3-8843-65c456a014e4" (UID: "abd2015c-3d81-47e3-8843-65c456a014e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:37.839522 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.839486    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd2015c-3d81-47e3-8843-65c456a014e4-kube-api-access-fmznk" (OuterVolumeSpecName: "kube-api-access-fmznk") pod "abd2015c-3d81-47e3-8843-65c456a014e4" (UID: "abd2015c-3d81-47e3-8843-65c456a014e4"). InnerVolumeSpecName "kube-api-access-fmznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:37.839795 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.839779    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd2015c-3d81-47e3-8843-65c456a014e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abd2015c-3d81-47e3-8843-65c456a014e4" (UID: "abd2015c-3d81-47e3-8843-65c456a014e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:37.856528 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.856500    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abd2015c-3d81-47e3-8843-65c456a014e4" (UID: "abd2015c-3d81-47e3-8843-65c456a014e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:37.897346 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.897318    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abd2015c-3d81-47e3-8843-65c456a014e4" (UID: "abd2015c-3d81-47e3-8843-65c456a014e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:37.938594 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.938575    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:05:37.938665 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.938595    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:05:37.938665 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.938609    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:05:37.938665 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.938619    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd2015c-3d81-47e3-8843-65c456a014e4-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:05:37.938665 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.938631    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmznk\" (UniqueName: \"kubernetes.io/projected/abd2015c-3d81-47e3-8843-65c456a014e4-kube-api-access-fmznk\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:05:37.938665 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:37.938639    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abd2015c-3d81-47e3-8843-65c456a014e4-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:05:38.339693 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.339660    2576 generic.go:356] "Generic (PLEG): container finished" podID="abd2015c-3d81-47e3-8843-65c456a014e4" containerID="63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2" exitCode=0
Jul 09 18:05:38.340137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.339716    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"
Jul 09 18:05:38.340137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.339743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" event={"ID":"abd2015c-3d81-47e3-8843-65c456a014e4","Type":"ContainerDied","Data":"63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2"}
Jul 09 18:05:38.340137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.339792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs" event={"ID":"abd2015c-3d81-47e3-8843-65c456a014e4","Type":"ContainerDied","Data":"5d4991f718ec818de8e1e0c041db038731dd920fb6c6729816ebbbab6edd9c80"}
Jul 09 18:05:38.340137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.339815    2576 scope.go:117] "RemoveContainer" containerID="63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2"
Jul 09 18:05:38.349021 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.349005    2576 scope.go:117] "RemoveContainer" containerID="6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92"
Jul 09 18:05:38.357094 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.357079    2576 scope.go:117] "RemoveContainer" containerID="63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2"
Jul 09 18:05:38.357323 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:05:38.357306    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2\": container with ID starting with 63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2 not found: ID does not exist" containerID="63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2"
Jul 09 18:05:38.357368 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.357332    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2"} err="failed to get container status \"63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2\": rpc error: code = NotFound desc = could not find container \"63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2\": container with ID starting with 63cfa073a837f3ce84f5cdf3f336d9a3100a373177568ca57992b38d2333f7f2 not found: ID does not exist"
Jul 09 18:05:38.357368 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.357350    2576 scope.go:117] "RemoveContainer" containerID="6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92"
Jul 09 18:05:38.357579 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:05:38.357562    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92\": container with ID starting with 6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92 not found: ID does not exist" containerID="6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92"
Jul 09 18:05:38.357626 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.357584    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92"} err="failed to get container status \"6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92\": rpc error: code = NotFound desc = could not find container \"6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92\": container with ID starting with 6f93b0473ca8edafcec5e8a992be9f4e28674ecd29a396cebd0c9cd010e32a92 not found: ID does not exist"
Jul 09 18:05:38.362577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.362556    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"]
Jul 09 18:05:38.368202 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:38.368181    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs"]
Jul 09 18:05:39.669944 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:05:39.669905    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" path="/var/lib/kubelet/pods/abd2015c-3d81-47e3-8843-65c456a014e4/volumes"
Jul 09 18:06:08.596808 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.596772    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"]
Jul 09 18:06:08.597431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.597145    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="storage-initializer"
Jul 09 18:06:08.597431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.597158    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="storage-initializer"
Jul 09 18:06:08.597431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.597188    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main"
Jul 09 18:06:08.597431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.597197    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main"
Jul 09 18:06:08.597431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.597283    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd2015c-3d81-47e3-8843-65c456a014e4" containerName="main"
Jul 09 18:06:08.606108 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.606081    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.608873 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.608847    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 09 18:06:08.612342 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.612318    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"]
Jul 09 18:06:08.707210 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.707182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.707210 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.707222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.707473 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.707243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.707473 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.707358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.707473 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.707405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.707473 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.707451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrq5\" (UniqueName: \"kubernetes.io/projected/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-kube-api-access-rtrq5\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808189 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrq5\" (UniqueName: \"kubernetes.io/projected/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-kube-api-access-rtrq5\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808666 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808535    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808728 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.808785 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.808750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.810410 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.810388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.810757 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.810739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:08.816923 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:08.816902    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrq5\" (UniqueName: \"kubernetes.io/projected/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-kube-api-access-rtrq5\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:12.742600 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:12.742565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:12.748855 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:12.748834    2576 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:12.843952 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:12.843918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:12.844131 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:12.844081    2576 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:12.847377 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:12.847360    2576 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:12.946408 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:12.946370    2576 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/17b411dca5d48cfac4cdaa10248b91d7ef13accf2569b96b481df6f6ce336a23/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:12.962328 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:12.962300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:13.117438 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:13.117402    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:13.254194 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:13.254162    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"]
Jul 09 18:06:13.255629 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:06:13.255600    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c740f0_f688_4e6d_bcb3_f21b0f0180b1.slice/crio-b372e8c97be1c91dfb44da8a302d5b992db5a0b4b58f415d3a6026bbebb182c8 WatchSource:0}: Error finding container b372e8c97be1c91dfb44da8a302d5b992db5a0b4b58f415d3a6026bbebb182c8: Status 404 returned error can't find the container with id b372e8c97be1c91dfb44da8a302d5b992db5a0b4b58f415d3a6026bbebb182c8
Jul 09 18:06:13.506256 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:13.506164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" event={"ID":"06c740f0-f688-4e6d-bcb3-f21b0f0180b1","Type":"ContainerStarted","Data":"16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589"}
Jul 09 18:06:13.506256 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:13.506216    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" event={"ID":"06c740f0-f688-4e6d-bcb3-f21b0f0180b1","Type":"ContainerStarted","Data":"b372e8c97be1c91dfb44da8a302d5b992db5a0b4b58f415d3a6026bbebb182c8"}
Jul 09 18:06:13.526161 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:13.526112    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podStartSLOduration=5.526095001 podStartE2EDuration="5.526095001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:13.52415507 +0000 UTC m=+1603.054932101" watchObservedRunningTime="2026-07-09 18:06:13.526095001 +0000 UTC m=+1603.056872030"
Jul 09 18:06:14.118118 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:14.118079    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:17.778667 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.778638    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"]
Jul 09 18:06:17.781561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.781538    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.784346 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.784317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 09 18:06:17.795637 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.795614    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"]
Jul 09 18:06:17.896294 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.896250    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa592ed2-da64-4911-8426-829c88c98044-tls-certs\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.896462 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.896371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-tmp-dir\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.896462 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.896408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-home\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.896462 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.896432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zgg7\" (UniqueName: \"kubernetes.io/projected/fa592ed2-da64-4911-8426-829c88c98044-kube-api-access-4zgg7\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.896462 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.896452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-model-cache\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.896600 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.896544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-dshm\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.896600 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.896570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997324 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-tmp-dir\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997324 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-home\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997546 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgg7\" (UniqueName: \"kubernetes.io/projected/fa592ed2-da64-4911-8426-829c88c98044-kube-api-access-4zgg7\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997546 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-model-cache\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997546 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-dshm\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997546 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997546 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa592ed2-da64-4911-8426-829c88c98044-tls-certs\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997818 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-tmp-dir\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997818 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997771    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-home\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.997927 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.997868    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-model-cache\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.998065 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.998021    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.999754 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.999735    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-dshm\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:17.999989 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:17.999974    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa592ed2-da64-4911-8426-829c88c98044-tls-certs\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:18.005767 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:18.005746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zgg7\" (UniqueName: \"kubernetes.io/projected/fa592ed2-da64-4911-8426-829c88c98044-kube-api-access-4zgg7\") pod \"custom-route-timeout-test-kserve-b964bcdd-z2m45\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:18.097280 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:18.097243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:18.241911 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:18.241879    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"]
Jul 09 18:06:18.243229 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:06:18.243206    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa592ed2_da64_4911_8426_829c88c98044.slice/crio-1ed881aef6a327b8f5f057a22346bee91c700f821a4c2e47e02d6827d243bf1d WatchSource:0}: Error finding container 1ed881aef6a327b8f5f057a22346bee91c700f821a4c2e47e02d6827d243bf1d: Status 404 returned error can't find the container with id 1ed881aef6a327b8f5f057a22346bee91c700f821a4c2e47e02d6827d243bf1d
Jul 09 18:06:18.535496 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:18.535402    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" event={"ID":"fa592ed2-da64-4911-8426-829c88c98044","Type":"ContainerStarted","Data":"917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29"}
Jul 09 18:06:18.535496 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:18.535459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" event={"ID":"fa592ed2-da64-4911-8426-829c88c98044","Type":"ContainerStarted","Data":"1ed881aef6a327b8f5f057a22346bee91c700f821a4c2e47e02d6827d243bf1d"}
Jul 09 18:06:22.560752 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:22.560713    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa592ed2-da64-4911-8426-829c88c98044" containerID="917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29" exitCode=0
Jul 09 18:06:22.561189 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:22.560790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" event={"ID":"fa592ed2-da64-4911-8426-829c88c98044","Type":"ContainerDied","Data":"917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29"}
Jul 09 18:06:23.118145 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:23.118103    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:06:23.119944 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:23.119915    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:06:23.567203 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:23.567162    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" event={"ID":"fa592ed2-da64-4911-8426-829c88c98044","Type":"ContainerStarted","Data":"d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8"}
Jul 09 18:06:23.602485 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:23.602422    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podStartSLOduration=3.1384054 podStartE2EDuration="6.602401564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.463996164s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:23.599990641 +0000 UTC m=+1613.130767674" watchObservedRunningTime="2026-07-09 18:06:23.602401564 +0000 UTC m=+1613.133178595"
Jul 09 18:06:24.098380 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:24.098342    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:28.097691 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:28.097651    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:06:28.099350 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:28.099317    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:06:33.118602 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:33.118555    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:06:38.097719 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:38.097666    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:06:43.118136 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:43.118093    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:06:48.098090 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:48.098018    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:06:53.118762 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:53.118716    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:06:58.098564 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:06:58.098517    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:07:03.118201 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:03.118158    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:07:08.098050 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:08.097996    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:07:13.118376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:13.118325    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:07:18.098027 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:18.097971    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:07:23.118616 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:23.118569    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:07:28.098413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:28.098368    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:07:33.118340 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:33.118295    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:07:38.097864 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:38.097821    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:07:43.118222 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:43.118180    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:07:48.098122 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:48.098072    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:07:53.118714 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:53.118619    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:07:58.097753 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:07:58.097711    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:08:03.118189 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:03.118146    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:08:08.097812 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:08.097770    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:08:13.117842 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:13.117796    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:08:18.098229 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:18.098190    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:08:23.117961 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:23.117918    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:08:28.098353 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:28.098312    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:08:33.118810 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:33.118763    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 09 18:08:38.098278 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:38.098221    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 09 18:08:43.128395 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:43.128364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:08:43.136673 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:43.136654    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:08:47.404277 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:47.404229    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"]
Jul 09 18:08:48.107778 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:48.107747    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:08:48.115811 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:48.115786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:08:54.200850 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:08:54.200818    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"]
Jul 09 18:09:02.429622 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:02.429584    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" containerID="cri-o://16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589" gracePeriod=45
Jul 09 18:09:03.172442 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.172419    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:09:03.201330 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201270    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") "
Jul 09 18:09:03.201330 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201313    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtrq5\" (UniqueName: \"kubernetes.io/projected/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-kube-api-access-rtrq5\") pod \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") "
Jul 09 18:09:03.201518 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201380    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tmp-dir\") pod \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") "
Jul 09 18:09:03.201518 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201423    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-home\") pod \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") "
Jul 09 18:09:03.201518 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201446    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-model-cache\") pod \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") "
Jul 09 18:09:03.201518 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201474    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-dshm\") pod \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") "
Jul 09 18:09:03.201725 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201519    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tls-certs\") pod \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\" (UID: \"06c740f0-f688-4e6d-bcb3-f21b0f0180b1\") "
Jul 09 18:09:03.201725 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201682    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "06c740f0-f688-4e6d-bcb3-f21b0f0180b1" (UID: "06c740f0-f688-4e6d-bcb3-f21b0f0180b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:03.201896 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.201871    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:03.202098 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.202075    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-home" (OuterVolumeSpecName: "home") pod "06c740f0-f688-4e6d-bcb3-f21b0f0180b1" (UID: "06c740f0-f688-4e6d-bcb3-f21b0f0180b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:03.204020 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.203989    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-dshm" (OuterVolumeSpecName: "dshm") pod "06c740f0-f688-4e6d-bcb3-f21b0f0180b1" (UID: "06c740f0-f688-4e6d-bcb3-f21b0f0180b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:03.204161 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.204029    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-kube-api-access-rtrq5" (OuterVolumeSpecName: "kube-api-access-rtrq5") pod "06c740f0-f688-4e6d-bcb3-f21b0f0180b1" (UID: "06c740f0-f688-4e6d-bcb3-f21b0f0180b1"). InnerVolumeSpecName "kube-api-access-rtrq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:09:03.204236 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.204192    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06c740f0-f688-4e6d-bcb3-f21b0f0180b1" (UID: "06c740f0-f688-4e6d-bcb3-f21b0f0180b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:09:03.205416 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.205393    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf" (OuterVolumeSpecName: "kserve-pvc-source") pod "06c740f0-f688-4e6d-bcb3-f21b0f0180b1" (UID: "06c740f0-f688-4e6d-bcb3-f21b0f0180b1"). InnerVolumeSpecName "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:09:03.220085 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.220055    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06c740f0-f688-4e6d-bcb3-f21b0f0180b1" (UID: "06c740f0-f688-4e6d-bcb3-f21b0f0180b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:03.303083 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.303060    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:03.303083 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.303080    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:03.303217 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.303089    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:03.303217 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.303117    2576 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-132-16.ec2.internal\" "
Jul 09 18:09:03.303217 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.303128    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrq5\" (UniqueName: \"kubernetes.io/projected/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-kube-api-access-rtrq5\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:03.303217 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.303137    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c740f0-f688-4e6d-bcb3-f21b0f0180b1-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:03.329113 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.329098    2576 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf") on node "ip-10-0-132-16.ec2.internal"
Jul 09 18:09:03.404163 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.404139    2576 reconciler_common.go:299] "Volume detached for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\""
Jul 09 18:09:03.415049 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.415019    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"
Jul 09 18:09:03.415135 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.415032    2576 generic.go:356] "Generic (PLEG): container finished" podID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerID="16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589" exitCode=0
Jul 09 18:09:03.415183 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.415060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" event={"ID":"06c740f0-f688-4e6d-bcb3-f21b0f0180b1","Type":"ContainerDied","Data":"16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589"}
Jul 09 18:09:03.415227 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.415184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" event={"ID":"06c740f0-f688-4e6d-bcb3-f21b0f0180b1","Type":"ContainerDied","Data":"b372e8c97be1c91dfb44da8a302d5b992db5a0b4b58f415d3a6026bbebb182c8"}
Jul 09 18:09:03.415227 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.415206    2576 scope.go:117] "RemoveContainer" containerID="16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589"
Jul 09 18:09:03.424071 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.424051    2576 scope.go:117] "RemoveContainer" containerID="16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589"
Jul 09 18:09:03.424369 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:09:03.424347    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589\": container with ID starting with 16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589 not found: ID does not exist" containerID="16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589"
Jul 09 18:09:03.424432 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.424372    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589"} err="failed to get container status \"16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589\": rpc error: code = NotFound desc = could not find container \"16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589\": container with ID starting with 16df9f678dc078b706398208c8623b3ca9f282a33ea8bbbea44ffb3c05c24589 not found: ID does not exist"
Jul 09 18:09:03.436156 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.436133    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"]
Jul 09 18:09:03.441784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.441763    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs"]
Jul 09 18:09:03.670070 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:03.670020    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" path="/var/lib/kubelet/pods/06c740f0-f688-4e6d-bcb3-f21b0f0180b1/volumes"
Jul 09 18:09:04.129842 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:04.129796    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dbbtfs" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:09:09.224998 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:09.224958    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main" containerID="cri-o://d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8" gracePeriod=45
Jul 09 18:09:09.989436 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:09.989412    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:09:10.055380 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.055316    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zgg7\" (UniqueName: \"kubernetes.io/projected/fa592ed2-da64-4911-8426-829c88c98044-kube-api-access-4zgg7\") pod \"fa592ed2-da64-4911-8426-829c88c98044\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") "
Jul 09 18:09:10.055380 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.055347    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-tmp-dir\") pod \"fa592ed2-da64-4911-8426-829c88c98044\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") "
Jul 09 18:09:10.055536 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.055403    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-kserve-provision-location\") pod \"fa592ed2-da64-4911-8426-829c88c98044\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") "
Jul 09 18:09:10.055536 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.055448    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-dshm\") pod \"fa592ed2-da64-4911-8426-829c88c98044\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") "
Jul 09 18:09:10.055536 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.055484    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa592ed2-da64-4911-8426-829c88c98044-tls-certs\") pod \"fa592ed2-da64-4911-8426-829c88c98044\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") "
Jul 09 18:09:10.055536 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.055520    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-home\") pod \"fa592ed2-da64-4911-8426-829c88c98044\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") "
Jul 09 18:09:10.055742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.055554    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-model-cache\") pod \"fa592ed2-da64-4911-8426-829c88c98044\" (UID: \"fa592ed2-da64-4911-8426-829c88c98044\") "
Jul 09 18:09:10.056110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.056079    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa592ed2-da64-4911-8426-829c88c98044" (UID: "fa592ed2-da64-4911-8426-829c88c98044"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:10.056289 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.056268    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-home" (OuterVolumeSpecName: "home") pod "fa592ed2-da64-4911-8426-829c88c98044" (UID: "fa592ed2-da64-4911-8426-829c88c98044"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:10.057490 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.057458    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa592ed2-da64-4911-8426-829c88c98044-kube-api-access-4zgg7" (OuterVolumeSpecName: "kube-api-access-4zgg7") pod "fa592ed2-da64-4911-8426-829c88c98044" (UID: "fa592ed2-da64-4911-8426-829c88c98044"). InnerVolumeSpecName "kube-api-access-4zgg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:09:10.057758 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.057726    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-dshm" (OuterVolumeSpecName: "dshm") pod "fa592ed2-da64-4911-8426-829c88c98044" (UID: "fa592ed2-da64-4911-8426-829c88c98044"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:10.057847 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.057790    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa592ed2-da64-4911-8426-829c88c98044-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa592ed2-da64-4911-8426-829c88c98044" (UID: "fa592ed2-da64-4911-8426-829c88c98044"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:09:10.069745 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.069724    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa592ed2-da64-4911-8426-829c88c98044" (UID: "fa592ed2-da64-4911-8426-829c88c98044"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:10.115224 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.115198    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa592ed2-da64-4911-8426-829c88c98044" (UID: "fa592ed2-da64-4911-8426-829c88c98044"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:10.157216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.157194    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:10.157216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.157215    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zgg7\" (UniqueName: \"kubernetes.io/projected/fa592ed2-da64-4911-8426-829c88c98044-kube-api-access-4zgg7\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:10.157359 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.157229    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:10.157359 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.157241    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:10.157359 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.157250    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:10.157359 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.157258    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa592ed2-da64-4911-8426-829c88c98044-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:10.157359 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.157265    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa592ed2-da64-4911-8426-829c88c98044-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:09:10.449306 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.449276    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa592ed2-da64-4911-8426-829c88c98044" containerID="d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8" exitCode=0
Jul 09 18:09:10.449760 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.449339    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"
Jul 09 18:09:10.449760 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.449368    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" event={"ID":"fa592ed2-da64-4911-8426-829c88c98044","Type":"ContainerDied","Data":"d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8"}
Jul 09 18:09:10.449760 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.449425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45" event={"ID":"fa592ed2-da64-4911-8426-829c88c98044","Type":"ContainerDied","Data":"1ed881aef6a327b8f5f057a22346bee91c700f821a4c2e47e02d6827d243bf1d"}
Jul 09 18:09:10.449760 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.449450    2576 scope.go:117] "RemoveContainer" containerID="d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8"
Jul 09 18:09:10.458827 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.458811    2576 scope.go:117] "RemoveContainer" containerID="917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29"
Jul 09 18:09:10.466472 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.466453    2576 scope.go:117] "RemoveContainer" containerID="d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8"
Jul 09 18:09:10.466709 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:09:10.466693    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8\": container with ID starting with d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8 not found: ID does not exist" containerID="d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8"
Jul 09 18:09:10.466753 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.466717    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8"} err="failed to get container status \"d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8\": rpc error: code = NotFound desc = could not find container \"d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8\": container with ID starting with d356be5990b73e655f4badc0572adc0ebb3a52565f937e299369c6a6beb54dc8 not found: ID does not exist"
Jul 09 18:09:10.466753 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.466732    2576 scope.go:117] "RemoveContainer" containerID="917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29"
Jul 09 18:09:10.466938 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:09:10.466922    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29\": container with ID starting with 917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29 not found: ID does not exist" containerID="917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29"
Jul 09 18:09:10.466975 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.466943    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29"} err="failed to get container status \"917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29\": rpc error: code = NotFound desc = could not find container \"917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29\": container with ID starting with 917226b6b71d05b4e34a5db0d484589a09514ce408a79eaff2c442ca430c2c29 not found: ID does not exist"
Jul 09 18:09:10.472460 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.472438    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"]
Jul 09 18:09:10.476734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:10.476706    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b964bcdd-z2m45"]
Jul 09 18:09:11.669760 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:11.669718    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa592ed2-da64-4911-8426-829c88c98044" path="/var/lib/kubelet/pods/fa592ed2-da64-4911-8426-829c88c98044/volumes"
Jul 09 18:09:12.614683 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.614651    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"]
Jul 09 18:09:12.615110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615090    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="storage-initializer"
Jul 09 18:09:12.615110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615107    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="storage-initializer"
Jul 09 18:09:12.615322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615139    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main"
Jul 09 18:09:12.615322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615146    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main"
Jul 09 18:09:12.615322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615153    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main"
Jul 09 18:09:12.615322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615159    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main"
Jul 09 18:09:12.615322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615222    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c740f0-f688-4e6d-bcb3-f21b0f0180b1" containerName="main"
Jul 09 18:09:12.615322 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.615231    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa592ed2-da64-4911-8426-829c88c98044" containerName="main"
Jul 09 18:09:12.620872 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.620711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.623861 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.623770    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 09 18:09:12.623861 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.623838    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-mzs4q\""
Jul 09 18:09:12.640742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.640719    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"]
Jul 09 18:09:12.642320 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.642300    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"]
Jul 09 18:09:12.645805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.645788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.656028 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.656009    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"]
Jul 09 18:09:12.677686 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.677657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.678066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.677692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gld\" (UniqueName: \"kubernetes.io/projected/7dd94724-050e-452e-b010-592f503139af-kube-api-access-26gld\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.678066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.677722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.678066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.677751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.678066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.677817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mq6x\" (UniqueName: \"kubernetes.io/projected/124f37a5-aba1-4e13-8540-9b188609a789-kube-api-access-7mq6x\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.678066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.677848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.678066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.678034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.678383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.678105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd94724-050e-452e-b010-592f503139af-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.678383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.678131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124f37a5-aba1-4e13-8540-9b188609a789-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.678383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.678157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.678383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.678205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.678383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.678231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779265 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779371 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd94724-050e-452e-b010-592f503139af-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.779371 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124f37a5-aba1-4e13-8540-9b188609a789-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779371 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.779487 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.779487 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779589 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.779643 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gld\" (UniqueName: \"kubernetes.io/projected/7dd94724-050e-452e-b010-592f503139af-kube-api-access-26gld\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.779703 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779703 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.779810 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mq6x\" (UniqueName: \"kubernetes.io/projected/124f37a5-aba1-4e13-8540-9b188609a789-kube-api-access-7mq6x\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779810 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779912 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.779912 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.779818    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.780070 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.780031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.780318 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.780288    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.780406 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.780325    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.780406 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.780358    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.782063 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.782025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.782257 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.782231    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.782321 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.782310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd94724-050e-452e-b010-592f503139af-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.782396 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.782378    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124f37a5-aba1-4e13-8540-9b188609a789-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:12.788186 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.788165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gld\" (UniqueName: \"kubernetes.io/projected/7dd94724-050e-452e-b010-592f503139af-kube-api-access-26gld\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:12.788580 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:12.788557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mq6x\" (UniqueName: \"kubernetes.io/projected/124f37a5-aba1-4e13-8540-9b188609a789-kube-api-access-7mq6x\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:15.705941 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.705911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:15.711622 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.711593    2576 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:15.807360 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.807333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:15.807470 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.807425    2576 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:15.810321 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.810298    2576 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:15.893541 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.893512    2576 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/17b411dca5d48cfac4cdaa10248b91d7ef13accf2569b96b481df6f6ce336a23/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:15.909283 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.909259    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:15.955738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:15.955716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:16.010021 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.009990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:16.010185 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.010157    2576 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:16.014256 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.014211    2576 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:16.032810 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.032788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:16.110323 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.110298    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"]
Jul 09 18:09:16.112079 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:09:16.112026    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd94724_050e_452e_b010_592f503139af.slice/crio-28a31580e1a78063c3f3cf9f75d610c0b2a6b994865e67914284ae94d224ee29 WatchSource:0}: Error finding container 28a31580e1a78063c3f3cf9f75d610c0b2a6b994865e67914284ae94d224ee29: Status 404 returned error can't find the container with id 28a31580e1a78063c3f3cf9f75d610c0b2a6b994865e67914284ae94d224ee29
Jul 09 18:09:16.231873 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.231790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:16.369332 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.369286    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"]
Jul 09 18:09:16.372576 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:09:16.372547    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124f37a5_aba1_4e13_8540_9b188609a789.slice/crio-39e7cb68f705bf20ca611400e3c532b24d656f74f07124cc9e81802a8af68dbb WatchSource:0}: Error finding container 39e7cb68f705bf20ca611400e3c532b24d656f74f07124cc9e81802a8af68dbb: Status 404 returned error can't find the container with id 39e7cb68f705bf20ca611400e3c532b24d656f74f07124cc9e81802a8af68dbb
Jul 09 18:09:16.374713 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.374689    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:09:16.481688 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.481651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" event={"ID":"7dd94724-050e-452e-b010-592f503139af","Type":"ContainerStarted","Data":"e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d"}
Jul 09 18:09:16.481912 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.481715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" event={"ID":"7dd94724-050e-452e-b010-592f503139af","Type":"ContainerStarted","Data":"28a31580e1a78063c3f3cf9f75d610c0b2a6b994865e67914284ae94d224ee29"}
Jul 09 18:09:16.482888 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.482865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" event={"ID":"124f37a5-aba1-4e13-8540-9b188609a789","Type":"ContainerStarted","Data":"39e7cb68f705bf20ca611400e3c532b24d656f74f07124cc9e81802a8af68dbb"}
Jul 09 18:09:16.501947 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:16.501888    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podStartSLOduration=4.501872465 podStartE2EDuration="4.501872465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:16.501453439 +0000 UTC m=+1786.032230465" watchObservedRunningTime="2026-07-09 18:09:16.501872465 +0000 UTC m=+1786.032649496"
Jul 09 18:09:17.491097 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:17.490988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" event={"ID":"124f37a5-aba1-4e13-8540-9b188609a789","Type":"ContainerStarted","Data":"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"}
Jul 09 18:09:17.491507 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:17.491295    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:18.498682 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:18.498647    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" event={"ID":"124f37a5-aba1-4e13-8540-9b188609a789","Type":"ContainerStarted","Data":"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"}
Jul 09 18:09:18.522674 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:18.522615    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podStartSLOduration=5.746723351 podStartE2EDuration="6.522594287s" totalImagesPullingTime="775.870936ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:18.520575549 +0000 UTC m=+1788.051352602" watchObservedRunningTime="2026-07-09 18:09:18.522594287 +0000 UTC m=+1788.053371317"
Jul 09 18:09:25.956475 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:25.956449    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:25.956824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:25.956501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:09:25.958142 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:25.958116    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:09:26.232244 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:26.232159    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:26.232244 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:26.232227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:26.233505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:26.233476    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:09:31.059281 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:31.059187    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:09:31.061235 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:31.061206    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:09:35.957005 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:35.956953    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:09:36.232472 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:36.232363    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:09:36.254383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:36.254345    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:09:45.956445 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:45.956393    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:09:46.232997 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:46.232889    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:09:55.956335 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:55.956290    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:09:56.233074 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:09:56.232947    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:10:04.099661 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.099618    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"]
Jul 09 18:10:04.103288 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.103265    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.106074 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.106028    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 09 18:10:04.114489 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.114467    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"]
Jul 09 18:10:04.133001 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.132974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-dshm\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.133142 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.133052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-model-cache\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.133209 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.133153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-home\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.133209 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.133194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tls-certs\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.133299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.133241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kserve-provision-location\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.133299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.133272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbk4\" (UniqueName: \"kubernetes.io/projected/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kube-api-access-xjbk4\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.133403 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.133343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tmp-dir\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.235854 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.235816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-dshm\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236022 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.235918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-model-cache\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236022 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.235957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-home\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236022 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.235982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tls-certs\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.236025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kserve-provision-location\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.236073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbk4\" (UniqueName: \"kubernetes.io/projected/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kube-api-access-xjbk4\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.236109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tmp-dir\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236447 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.236417    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-model-cache\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236638 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.236615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-home\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236730 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.236703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kserve-provision-location\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.236730 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.236476    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tmp-dir\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.238396 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.238377    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-dshm\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.238929 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.238911    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tls-certs\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.247213 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.247186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbk4\" (UniqueName: \"kubernetes.io/projected/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kube-api-access-xjbk4\") pod \"router-with-refs-test-kserve-5f8f4fd684-jp6nc\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.414230 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.414147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:04.573245 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.573217    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"]
Jul 09 18:10:04.575449 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:10:04.575418    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81daaaa_5d76_4784_8dcf_aaab28b8df0c.slice/crio-2c7eff1da5a9fb671377573f1373bbf7841b6fddb80f3f3b64e69b9a3bd31cb8 WatchSource:0}: Error finding container 2c7eff1da5a9fb671377573f1373bbf7841b6fddb80f3f3b64e69b9a3bd31cb8: Status 404 returned error can't find the container with id 2c7eff1da5a9fb671377573f1373bbf7841b6fddb80f3f3b64e69b9a3bd31cb8
Jul 09 18:10:04.766214 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.766094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" event={"ID":"b81daaaa-5d76-4784-8dcf-aaab28b8df0c","Type":"ContainerStarted","Data":"8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3"}
Jul 09 18:10:04.766429 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:04.766400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" event={"ID":"b81daaaa-5d76-4784-8dcf-aaab28b8df0c","Type":"ContainerStarted","Data":"2c7eff1da5a9fb671377573f1373bbf7841b6fddb80f3f3b64e69b9a3bd31cb8"}
Jul 09 18:10:05.956214 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:05.956160    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:10:06.233316 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:06.233205    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:10:08.801678 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:08.801645    2576 generic.go:356] "Generic (PLEG): container finished" podID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerID="8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3" exitCode=0
Jul 09 18:10:08.802122 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:08.801720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" event={"ID":"b81daaaa-5d76-4784-8dcf-aaab28b8df0c","Type":"ContainerDied","Data":"8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3"}
Jul 09 18:10:09.811504 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:09.811458    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" event={"ID":"b81daaaa-5d76-4784-8dcf-aaab28b8df0c","Type":"ContainerStarted","Data":"e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8"}
Jul 09 18:10:09.836609 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:09.836537    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podStartSLOduration=1.82551738 podStartE2EDuration="5.836515013s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.010997633s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:09.834887323 +0000 UTC m=+1839.365664377" watchObservedRunningTime="2026-07-09 18:10:09.836515013 +0000 UTC m=+1839.367292044"
Jul 09 18:10:10.415077 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:10.415017    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:14.415129 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:14.415089    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:10:14.417071 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:14.417028    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:10:15.957004 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:15.956955    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:10:16.232864 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:16.232769    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:10:24.414737 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:24.414691    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:10:25.956395 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:25.956343    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:10:26.232334 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:26.232235    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:10:34.414922 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:34.414883    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:10:35.956260 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:35.956219    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:10:36.232706 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:36.232602    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:10:44.415675 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:44.415617    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:10:45.956640 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:45.956597    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:10:46.232216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:46.232121    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:10:54.415227 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:54.415180    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:10:55.956263 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:55.956221    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:10:56.232730 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:10:56.232592    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:11:04.415001 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:04.414949    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:11:05.956960 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:05.956914    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:11:06.232441 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:06.232334    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:11:14.415246 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:14.415196    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:11:15.956826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:15.956780    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:11:16.232229 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:16.232124    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:11:24.414719 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:24.414669    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:11:25.956979 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:25.956929    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:11:26.233027 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:26.232924    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Jul 09 18:11:34.415295 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:34.415257    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:11:35.956599 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:35.956559    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:11:36.247193 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:36.247116    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:11:36.259093 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:36.259066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:11:44.415352 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:44.415313    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:11:45.956033 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:45.955998    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:11:54.414563 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:54.414513    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:11:55.956580 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:11:55.956536    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:12:04.414759 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:04.414710    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:12:05.956765 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:05.956718    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:12:14.415029 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:14.414984    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:12:15.956683 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:15.956632    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:12:24.424748 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:24.424666    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:12:24.432524 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:24.432500    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:12:25.956781 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:25.956744    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 09 18:12:31.210782 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:31.210747    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"]
Jul 09 18:12:35.966016 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:35.965986    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:12:35.973959 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:35.973930    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:12:40.880803 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:40.880768    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"]
Jul 09 18:12:40.885368 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:40.885337    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"]
Jul 09 18:12:46.244684 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:46.244646    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" containerID="cri-o://e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8" gracePeriod=45
Jul 09 18:12:46.425698 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:46.425664    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 09 18:12:47.002011 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.001951    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:12:47.080935 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.080910    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-model-cache\") pod \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") "
Jul 09 18:12:47.081116 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.080942    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kserve-provision-location\") pod \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") "
Jul 09 18:12:47.081116 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.080978    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-home\") pod \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") "
Jul 09 18:12:47.081116 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.081111    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tls-certs\") pod \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") "
Jul 09 18:12:47.081387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.081158    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjbk4\" (UniqueName: \"kubernetes.io/projected/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kube-api-access-xjbk4\") pod \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") "
Jul 09 18:12:47.081387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.081183    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tmp-dir\") pod \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") "
Jul 09 18:12:47.081387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.081207    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-dshm\") pod \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\" (UID: \"b81daaaa-5d76-4784-8dcf-aaab28b8df0c\") "
Jul 09 18:12:47.081821 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.081636    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b81daaaa-5d76-4784-8dcf-aaab28b8df0c" (UID: "b81daaaa-5d76-4784-8dcf-aaab28b8df0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.081821 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.081756    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-home" (OuterVolumeSpecName: "home") pod "b81daaaa-5d76-4784-8dcf-aaab28b8df0c" (UID: "b81daaaa-5d76-4784-8dcf-aaab28b8df0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.083824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.083781    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kube-api-access-xjbk4" (OuterVolumeSpecName: "kube-api-access-xjbk4") pod "b81daaaa-5d76-4784-8dcf-aaab28b8df0c" (UID: "b81daaaa-5d76-4784-8dcf-aaab28b8df0c"). InnerVolumeSpecName "kube-api-access-xjbk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:47.084028 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.084007    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b81daaaa-5d76-4784-8dcf-aaab28b8df0c" (UID: "b81daaaa-5d76-4784-8dcf-aaab28b8df0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:12:47.084184 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.084156    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-dshm" (OuterVolumeSpecName: "dshm") pod "b81daaaa-5d76-4784-8dcf-aaab28b8df0c" (UID: "b81daaaa-5d76-4784-8dcf-aaab28b8df0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.094840 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.094815    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b81daaaa-5d76-4784-8dcf-aaab28b8df0c" (UID: "b81daaaa-5d76-4784-8dcf-aaab28b8df0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.148506 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.148469    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b81daaaa-5d76-4784-8dcf-aaab28b8df0c" (UID: "b81daaaa-5d76-4784-8dcf-aaab28b8df0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:47.182480 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.182460    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjbk4\" (UniqueName: \"kubernetes.io/projected/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kube-api-access-xjbk4\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.182480 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.182482    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.182608 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.182492    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.182608 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.182500    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.182608 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.182509    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.182608 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.182517    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.182608 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.182524    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81daaaa-5d76-4784-8dcf-aaab28b8df0c-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:47.721167 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.721136    2576 generic.go:356] "Generic (PLEG): container finished" podID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerID="e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8" exitCode=0
Jul 09 18:12:47.721577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.721179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" event={"ID":"b81daaaa-5d76-4784-8dcf-aaab28b8df0c","Type":"ContainerDied","Data":"e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8"}
Jul 09 18:12:47.721577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.721201    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"
Jul 09 18:12:47.721577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.721231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc" event={"ID":"b81daaaa-5d76-4784-8dcf-aaab28b8df0c","Type":"ContainerDied","Data":"2c7eff1da5a9fb671377573f1373bbf7841b6fddb80f3f3b64e69b9a3bd31cb8"}
Jul 09 18:12:47.721577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.721253    2576 scope.go:117] "RemoveContainer" containerID="e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8"
Jul 09 18:12:47.731081 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.731063    2576 scope.go:117] "RemoveContainer" containerID="8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3"
Jul 09 18:12:47.738788 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.738762    2576 scope.go:117] "RemoveContainer" containerID="e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8"
Jul 09 18:12:47.739082 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:12:47.739055    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8\": container with ID starting with e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8 not found: ID does not exist" containerID="e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8"
Jul 09 18:12:47.739203 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.739087    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8"} err="failed to get container status \"e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8\": rpc error: code = NotFound desc = could not find container \"e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8\": container with ID starting with e9ed3e67fb66498f0c39800b14f2e84232fa59ac8aec976ba12a3fe3c53a5df8 not found: ID does not exist"
Jul 09 18:12:47.739203 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.739105    2576 scope.go:117] "RemoveContainer" containerID="8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3"
Jul 09 18:12:47.739430 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:12:47.739403    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3\": container with ID starting with 8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3 not found: ID does not exist" containerID="8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3"
Jul 09 18:12:47.739502 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.739429    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3"} err="failed to get container status \"8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3\": rpc error: code = NotFound desc = could not find container \"8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3\": container with ID starting with 8d7ea563fc3f1b92e5b2efbc0d721567587e89abe9fb7cc3b9b624fcdf3407c3 not found: ID does not exist"
Jul 09 18:12:47.740685 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.740667    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"]
Jul 09 18:12:47.745212 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:47.745192    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5f8f4fd684-jp6nc"]
Jul 09 18:12:49.669438 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:49.669405    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" path="/var/lib/kubelet/pods/b81daaaa-5d76-4784-8dcf-aaab28b8df0c/volumes"
Jul 09 18:12:53.845316 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.845284    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"]
Jul 09 18:12:53.845694 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.845659    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="storage-initializer"
Jul 09 18:12:53.845694 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.845670    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="storage-initializer"
Jul 09 18:12:53.845694 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.845683    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main"
Jul 09 18:12:53.845694 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.845688    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main"
Jul 09 18:12:53.845843 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.845748    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b81daaaa-5d76-4784-8dcf-aaab28b8df0c" containerName="main"
Jul 09 18:12:53.851294 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.851270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:53.854133 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.854094    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 09 18:12:53.854290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.854147    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-nwhbn\""
Jul 09 18:12:53.863028 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.863004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"]
Jul 09 18:12:53.869508 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.869482    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"]
Jul 09 18:12:53.873310 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.873291    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.887257 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.887238    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"]
Jul 09 18:12:53.938534 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.938623 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:53.938623 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc9b060-bcca-457a-a28d-1cb65e569db6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:53.938623 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.938744 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:53.938744 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:53.938744 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:53.938873 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974nc\" (UniqueName: \"kubernetes.io/projected/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kube-api-access-974nc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.938873 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.938873 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.938997 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.938997 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx7hx\" (UniqueName: \"kubernetes.io/projected/1cc9b060-bcca-457a-a28d-1cb65e569db6-kube-api-access-zx7hx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:53.938997 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:53.939141 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:53.938999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.039535 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.039642 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.039642 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.039642 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc9b060-bcca-457a-a28d-1cb65e569db6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.039642 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.039642 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.039918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.039918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.039918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974nc\" (UniqueName: \"kubernetes.io/projected/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kube-api-access-974nc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.039918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.039918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.039918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx7hx\" (UniqueName: \"kubernetes.io/projected/1cc9b060-bcca-457a-a28d-1cb65e569db6-kube-api-access-zx7hx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039983    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.040005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.040074    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.039977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.040258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.040238    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.040569 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.040378    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.040613 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.040589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.042361 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.042338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc9b060-bcca-457a-a28d-1cb65e569db6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.042466 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.042421    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.042659 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.042638    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.042857 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.042843    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.048317 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.048299    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-974nc\" (UniqueName: \"kubernetes.io/projected/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kube-api-access-974nc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.048579 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.048558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx7hx\" (UniqueName: \"kubernetes.io/projected/1cc9b060-bcca-457a-a28d-1cb65e569db6-kube-api-access-zx7hx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.162455 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.162386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:54.184400 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.184379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:12:54.325966 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.325942    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"]
Jul 09 18:12:54.327709 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:12:54.327685    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc9b060_bcca_457a_a28d_1cb65e569db6.slice/crio-df9becc0564ca9af6d7be9ce48fb76a32642987ae80682d6b3e87b590437b6b2 WatchSource:0}: Error finding container df9becc0564ca9af6d7be9ce48fb76a32642987ae80682d6b3e87b590437b6b2: Status 404 returned error can't find the container with id df9becc0564ca9af6d7be9ce48fb76a32642987ae80682d6b3e87b590437b6b2
Jul 09 18:12:54.343549 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.343525    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"]
Jul 09 18:12:54.345703 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:12:54.345678    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e20d3a_34ab_4b17_883d_fb4dfeafaf7f.slice/crio-5b4fd89c25451686197bcad553d8c1308e24151f9879838c4ec3dced95efa0a2 WatchSource:0}: Error finding container 5b4fd89c25451686197bcad553d8c1308e24151f9879838c4ec3dced95efa0a2: Status 404 returned error can't find the container with id 5b4fd89c25451686197bcad553d8c1308e24151f9879838c4ec3dced95efa0a2
Jul 09 18:12:54.757314 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.757274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" event={"ID":"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f","Type":"ContainerStarted","Data":"191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d"}
Jul 09 18:12:54.757486 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.757330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" event={"ID":"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f","Type":"ContainerStarted","Data":"5b4fd89c25451686197bcad553d8c1308e24151f9879838c4ec3dced95efa0a2"}
Jul 09 18:12:54.758821 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.758799    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerStarted","Data":"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"}
Jul 09 18:12:54.758949 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.758844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerStarted","Data":"df9becc0564ca9af6d7be9ce48fb76a32642987ae80682d6b3e87b590437b6b2"}
Jul 09 18:12:54.758949 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:54.758883    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:12:55.767050 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:55.766997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerStarted","Data":"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"}
Jul 09 18:12:55.914520 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:55.914470    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main" containerID="cri-o://433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818" gracePeriod=45
Jul 09 18:12:55.914685 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:55.914612    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main" containerID="cri-o://e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d" gracePeriod=45
Jul 09 18:12:56.827420 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:56.827370    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="llm-d-routing-sidecar" containerID="cri-o://98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b" gracePeriod=45
Jul 09 18:12:57.025474 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.025377    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:12:57.118159 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.118132    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:12:57.179696 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.179655    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-model-cache\") pod \"7dd94724-050e-452e-b010-592f503139af\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") "
Jul 09 18:12:57.179696 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.179751    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-dshm\") pod \"7dd94724-050e-452e-b010-592f503139af\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") "
Jul 09 18:12:57.180180 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.179799    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-home\") pod \"7dd94724-050e-452e-b010-592f503139af\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") "
Jul 09 18:12:57.180180 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.179945    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"7dd94724-050e-452e-b010-592f503139af\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") "
Jul 09 18:12:57.180180 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.179969    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-model-cache" (OuterVolumeSpecName: "model-cache") pod "7dd94724-050e-452e-b010-592f503139af" (UID: "7dd94724-050e-452e-b010-592f503139af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.180180 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.179999    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26gld\" (UniqueName: \"kubernetes.io/projected/7dd94724-050e-452e-b010-592f503139af-kube-api-access-26gld\") pod \"7dd94724-050e-452e-b010-592f503139af\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") "
Jul 09 18:12:57.180180 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.180059    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd94724-050e-452e-b010-592f503139af-tls-certs\") pod \"7dd94724-050e-452e-b010-592f503139af\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") "
Jul 09 18:12:57.180747 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.180429    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-tmp-dir\") pod \"7dd94724-050e-452e-b010-592f503139af\" (UID: \"7dd94724-050e-452e-b010-592f503139af\") "
Jul 09 18:12:57.180747 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.180577    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-home" (OuterVolumeSpecName: "home") pod "7dd94724-050e-452e-b010-592f503139af" (UID: "7dd94724-050e-452e-b010-592f503139af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.182437 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.182414    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.182579 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.182567    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.183223 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.183199    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-dshm" (OuterVolumeSpecName: "dshm") pod "7dd94724-050e-452e-b010-592f503139af" (UID: "7dd94724-050e-452e-b010-592f503139af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.183568 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.183545    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd94724-050e-452e-b010-592f503139af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7dd94724-050e-452e-b010-592f503139af" (UID: "7dd94724-050e-452e-b010-592f503139af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:12:57.183649 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.183602    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd94724-050e-452e-b010-592f503139af-kube-api-access-26gld" (OuterVolumeSpecName: "kube-api-access-26gld") pod "7dd94724-050e-452e-b010-592f503139af" (UID: "7dd94724-050e-452e-b010-592f503139af"). InnerVolumeSpecName "kube-api-access-26gld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:57.184110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.184093    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf" (OuterVolumeSpecName: "kserve-pvc-source") pod "7dd94724-050e-452e-b010-592f503139af" (UID: "7dd94724-050e-452e-b010-592f503139af"). InnerVolumeSpecName "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:12:57.194125 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.194101    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7dd94724-050e-452e-b010-592f503139af" (UID: "7dd94724-050e-452e-b010-592f503139af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.283164 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283092    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"124f37a5-aba1-4e13-8540-9b188609a789\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") "
Jul 09 18:12:57.283164 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283151    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124f37a5-aba1-4e13-8540-9b188609a789-tls-certs\") pod \"124f37a5-aba1-4e13-8540-9b188609a789\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") "
Jul 09 18:12:57.283376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283188    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-model-cache\") pod \"124f37a5-aba1-4e13-8540-9b188609a789\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") "
Jul 09 18:12:57.283376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283248    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-tmp-dir\") pod \"124f37a5-aba1-4e13-8540-9b188609a789\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") "
Jul 09 18:12:57.283376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283276    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mq6x\" (UniqueName: \"kubernetes.io/projected/124f37a5-aba1-4e13-8540-9b188609a789-kube-api-access-7mq6x\") pod \"124f37a5-aba1-4e13-8540-9b188609a789\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") "
Jul 09 18:12:57.283376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-dshm\") pod \"124f37a5-aba1-4e13-8540-9b188609a789\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") "
Jul 09 18:12:57.283585 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283394    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-home\") pod \"124f37a5-aba1-4e13-8540-9b188609a789\" (UID: \"124f37a5-aba1-4e13-8540-9b188609a789\") "
Jul 09 18:12:57.283585 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283492    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-model-cache" (OuterVolumeSpecName: "model-cache") pod "124f37a5-aba1-4e13-8540-9b188609a789" (UID: "124f37a5-aba1-4e13-8540-9b188609a789"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.283880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283769    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.283880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283796    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26gld\" (UniqueName: \"kubernetes.io/projected/7dd94724-050e-452e-b010-592f503139af-kube-api-access-26gld\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.283880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283811    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.283880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283824    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd94724-050e-452e-b010-592f503139af-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.283880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.283835    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd94724-050e-452e-b010-592f503139af-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.284290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.284097    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-home" (OuterVolumeSpecName: "home") pod "124f37a5-aba1-4e13-8540-9b188609a789" (UID: "124f37a5-aba1-4e13-8540-9b188609a789"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.285810 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.285787    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-dshm" (OuterVolumeSpecName: "dshm") pod "124f37a5-aba1-4e13-8540-9b188609a789" (UID: "124f37a5-aba1-4e13-8540-9b188609a789"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.286137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.286012    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124f37a5-aba1-4e13-8540-9b188609a789-kube-api-access-7mq6x" (OuterVolumeSpecName: "kube-api-access-7mq6x") pod "124f37a5-aba1-4e13-8540-9b188609a789" (UID: "124f37a5-aba1-4e13-8540-9b188609a789"). InnerVolumeSpecName "kube-api-access-7mq6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:57.286137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.286097    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124f37a5-aba1-4e13-8540-9b188609a789-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "124f37a5-aba1-4e13-8540-9b188609a789" (UID: "124f37a5-aba1-4e13-8540-9b188609a789"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:12:57.286782 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.286762    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf" (OuterVolumeSpecName: "kserve-pvc-source") pod "124f37a5-aba1-4e13-8540-9b188609a789" (UID: "124f37a5-aba1-4e13-8540-9b188609a789"). InnerVolumeSpecName "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:12:57.295161 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.295138    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "124f37a5-aba1-4e13-8540-9b188609a789" (UID: "124f37a5-aba1-4e13-8540-9b188609a789"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:57.384975 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.384944    2576 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-132-16.ec2.internal\" "
Jul 09 18:12:57.384975 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.384971    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124f37a5-aba1-4e13-8540-9b188609a789-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.385220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.384983    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.385220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.384992    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mq6x\" (UniqueName: \"kubernetes.io/projected/124f37a5-aba1-4e13-8540-9b188609a789-kube-api-access-7mq6x\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.385220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.385000    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.385220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.385008    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124f37a5-aba1-4e13-8540-9b188609a789-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.412919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.412898    2576 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf") on node "ip-10-0-132-16.ec2.internal"
Jul 09 18:12:57.485791 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.485748    2576 reconciler_common.go:299] "Volume detached for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\""
Jul 09 18:12:57.781657 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.781623    2576 generic.go:356] "Generic (PLEG): container finished" podID="7dd94724-050e-452e-b010-592f503139af" containerID="e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d" exitCode=0
Jul 09 18:12:57.781805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.781666    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"
Jul 09 18:12:57.781805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.781713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" event={"ID":"7dd94724-050e-452e-b010-592f503139af","Type":"ContainerDied","Data":"e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d"}
Jul 09 18:12:57.781805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.781767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x" event={"ID":"7dd94724-050e-452e-b010-592f503139af","Type":"ContainerDied","Data":"28a31580e1a78063c3f3cf9f75d610c0b2a6b994865e67914284ae94d224ee29"}
Jul 09 18:12:57.781805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.781794    2576 scope.go:117] "RemoveContainer" containerID="e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d"
Jul 09 18:12:57.783558 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.783481    2576 generic.go:356] "Generic (PLEG): container finished" podID="124f37a5-aba1-4e13-8540-9b188609a789" containerID="433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818" exitCode=0
Jul 09 18:12:57.783558 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.783495    2576 generic.go:356] "Generic (PLEG): container finished" podID="124f37a5-aba1-4e13-8540-9b188609a789" containerID="98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b" exitCode=0
Jul 09 18:12:57.783558 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.783513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" event={"ID":"124f37a5-aba1-4e13-8540-9b188609a789","Type":"ContainerDied","Data":"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"}
Jul 09 18:12:57.783558 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.783552    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"
Jul 09 18:12:57.783775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.783569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" event={"ID":"124f37a5-aba1-4e13-8540-9b188609a789","Type":"ContainerDied","Data":"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"}
Jul 09 18:12:57.783775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.783594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv" event={"ID":"124f37a5-aba1-4e13-8540-9b188609a789","Type":"ContainerDied","Data":"39e7cb68f705bf20ca611400e3c532b24d656f74f07124cc9e81802a8af68dbb"}
Jul 09 18:12:57.796681 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.796665    2576 scope.go:117] "RemoveContainer" containerID="e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d"
Jul 09 18:12:57.796949 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:12:57.796923    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d\": container with ID starting with e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d not found: ID does not exist" containerID="e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d"
Jul 09 18:12:57.797003 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.796956    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d"} err="failed to get container status \"e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d\": rpc error: code = NotFound desc = could not find container \"e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d\": container with ID starting with e1013b1d6c1a2a0cb72fef77243b0c7b055d6fda4d8ef99b171f98f41fe8420d not found: ID does not exist"
Jul 09 18:12:57.797003 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.796975    2576 scope.go:117] "RemoveContainer" containerID="433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"
Jul 09 18:12:57.804373 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.804353    2576 scope.go:117] "RemoveContainer" containerID="98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"
Jul 09 18:12:57.808447 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.808426    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"]
Jul 09 18:12:57.812443 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.812425    2576 scope.go:117] "RemoveContainer" containerID="433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"
Jul 09 18:12:57.812701 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:12:57.812684    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818\": container with ID starting with 433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818 not found: ID does not exist" containerID="433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"
Jul 09 18:12:57.812758 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.812710    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"} err="failed to get container status \"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818\": rpc error: code = NotFound desc = could not find container \"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818\": container with ID starting with 433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818 not found: ID does not exist"
Jul 09 18:12:57.812758 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.812726    2576 scope.go:117] "RemoveContainer" containerID="98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"
Jul 09 18:12:57.812954 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:12:57.812936    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b\": container with ID starting with 98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b not found: ID does not exist" containerID="98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"
Jul 09 18:12:57.813025 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.812962    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"} err="failed to get container status \"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b\": rpc error: code = NotFound desc = could not find container \"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b\": container with ID starting with 98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b not found: ID does not exist"
Jul 09 18:12:57.813025 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.812983    2576 scope.go:117] "RemoveContainer" containerID="433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"
Jul 09 18:12:57.813237 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.813219    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818"} err="failed to get container status \"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818\": rpc error: code = NotFound desc = could not find container \"433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818\": container with ID starting with 433121e89f58ae6a78bf43cf581748c92bc236c43f4322675aeb92b1e6641818 not found: ID does not exist"
Jul 09 18:12:57.813308 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.813238    2576 scope.go:117] "RemoveContainer" containerID="98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"
Jul 09 18:12:57.813455 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.813432    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b"} err="failed to get container status \"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b\": rpc error: code = NotFound desc = could not find container \"98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b\": container with ID starting with 98f897b29593a79128da960f2290aa9b106658644275f0a09c22270c132e0a0b not found: ID does not exist"
Jul 09 18:12:57.813791 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.813775    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-d7xqv"]
Jul 09 18:12:57.838372 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.838318    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"]
Jul 09 18:12:57.840575 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:57.840548    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79vbf2x"]
Jul 09 18:12:59.669764 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:59.669732    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124f37a5-aba1-4e13-8540-9b188609a789" path="/var/lib/kubelet/pods/124f37a5-aba1-4e13-8540-9b188609a789/volumes"
Jul 09 18:12:59.670222 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:59.670208    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd94724-050e-452e-b010-592f503139af" path="/var/lib/kubelet/pods/7dd94724-050e-452e-b010-592f503139af/volumes"
Jul 09 18:12:59.797626 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:59.797599    2576 generic.go:356] "Generic (PLEG): container finished" podID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerID="191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d" exitCode=0
Jul 09 18:12:59.797761 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:59.797675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" event={"ID":"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f","Type":"ContainerDied","Data":"191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d"}
Jul 09 18:12:59.799456 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:59.799432    2576 generic.go:356] "Generic (PLEG): container finished" podID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerID="a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3" exitCode=0
Jul 09 18:12:59.799528 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:12:59.799503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerDied","Data":"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"}
Jul 09 18:13:00.806727 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:00.806684    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" event={"ID":"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f","Type":"ContainerStarted","Data":"723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee"}
Jul 09 18:13:00.809076 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:00.809048    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerStarted","Data":"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"}
Jul 09 18:13:00.829078 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:00.829002    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podStartSLOduration=3.293952624 podStartE2EDuration="7.828981596s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.535028972s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:00.827422089 +0000 UTC m=+2010.358199173" watchObservedRunningTime="2026-07-09 18:13:00.828981596 +0000 UTC m=+2010.359758627"
Jul 09 18:13:00.856939 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:00.856881    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podStartSLOduration=3.76983122 podStartE2EDuration="7.856859226s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.087028006s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:00.855252616 +0000 UTC m=+2010.386029674" watchObservedRunningTime="2026-07-09 18:13:00.856859226 +0000 UTC m=+2010.387636258"
Jul 09 18:13:04.163582 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:04.163541    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:13:04.164024 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:04.163645    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:13:04.164899 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:04.164859    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:13:04.182461 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:04.182431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:13:04.185356 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:04.185335    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:13:04.185470 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:04.185388    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:13:04.186622 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:04.186595    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:13:14.163557 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:14.163514    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:13:14.184717 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:14.184683    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:13:21.417052 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.416999    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:13:21.417594 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417574    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main"
Jul 09 18:13:21.417684 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417596    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main"
Jul 09 18:13:21.417684 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417614    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main"
Jul 09 18:13:21.417684 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417623    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main"
Jul 09 18:13:21.417684 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417649    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="llm-d-routing-sidecar"
Jul 09 18:13:21.417684 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417658    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="llm-d-routing-sidecar"
Jul 09 18:13:21.417925 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417749    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd94724-050e-452e-b010-592f503139af" containerName="main"
Jul 09 18:13:21.417925 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417763    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="llm-d-routing-sidecar"
Jul 09 18:13:21.417925 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.417777    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="124f37a5-aba1-4e13-8540-9b188609a789" containerName="main"
Jul 09 18:13:21.420202 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.420179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.426114 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.426086    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-4dl4v\""
Jul 09 18:13:21.427546 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.427523    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 09 18:13:21.445063 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.445001    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:13:21.478950 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.478912    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:13:21.481904 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.481882    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.496453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.496433    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:13:21.539300 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.539264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.539453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.539317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.539453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.539345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.539453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.539412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.539627 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.539469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69f701a-109d-44f3-8827-3c5c6ea7496d-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.539627 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.539556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6crz\" (UniqueName: \"kubernetes.io/projected/f69f701a-109d-44f3-8827-3c5c6ea7496d-kube-api-access-j6crz\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640210 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.640401 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640401 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640401 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640401 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.640625 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dq7c\" (UniqueName: \"kubernetes.io/projected/ffd15146-1c07-4caa-ac12-beadc5506dfe-kube-api-access-2dq7c\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.640625 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.640625 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640625 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69f701a-109d-44f3-8827-3c5c6ea7496d-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640835 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6crz\" (UniqueName: \"kubernetes.io/projected/f69f701a-109d-44f3-8827-3c5c6ea7496d-kube-api-access-j6crz\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640835 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640727    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.640936 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.641026 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.640710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.641160 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.641140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd15146-1c07-4caa-ac12-beadc5506dfe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.641307 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.641287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.643097 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.643076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.643206 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.643158    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69f701a-109d-44f3-8827-3c5c6ea7496d-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.652337 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.652317    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6crz\" (UniqueName: \"kubernetes.io/projected/f69f701a-109d-44f3-8827-3c5c6ea7496d-kube-api-access-j6crz\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:21.742099 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742099 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd15146-1c07-4caa-ac12-beadc5506dfe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742326 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dq7c\" (UniqueName: \"kubernetes.io/projected/ffd15146-1c07-4caa-ac12-beadc5506dfe-kube-api-access-2dq7c\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742494 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742494 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742494 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742457    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.742662 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.742640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.744413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.744387    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.744614 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.744594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd15146-1c07-4caa-ac12-beadc5506dfe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:21.750918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:21.750892    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dq7c\" (UniqueName: \"kubernetes.io/projected/ffd15146-1c07-4caa-ac12-beadc5506dfe-kube-api-access-2dq7c\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:24.162712 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:24.162659    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:13:24.185186 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:24.185090    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:13:25.799753 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:25.799708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:25.806332 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:25.806308    2576 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:25.900498 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:25.900460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:25.900669 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:25.900641    2576 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:25.904153 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:25.904132    2576 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:25.993347 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:25.993314    2576 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/17b411dca5d48cfac4cdaa10248b91d7ef13accf2569b96b481df6f6ce336a23/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:26.006166 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.006137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:26.101836 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.101792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:26.102033 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.101933    2576 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:26.105891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.105860    2576 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:26.120220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.120189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:26.230754 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.230712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:26.293007 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.292973    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:13:26.407184 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.407160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:13:26.472738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.472714    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:13:26.474158 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:13:26.474119    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd15146_1c07_4caa_ac12_beadc5506dfe.slice/crio-b3aa8b3ce155b2b2e2ebcc2172ebf05c276210e7c75f822b36aa40cab8c93f2c WatchSource:0}: Error finding container b3aa8b3ce155b2b2e2ebcc2172ebf05c276210e7c75f822b36aa40cab8c93f2c: Status 404 returned error can't find the container with id b3aa8b3ce155b2b2e2ebcc2172ebf05c276210e7c75f822b36aa40cab8c93f2c
Jul 09 18:13:26.961923 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.961673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ffd15146-1c07-4caa-ac12-beadc5506dfe","Type":"ContainerStarted","Data":"7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31"}
Jul 09 18:13:26.961923 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.961744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ffd15146-1c07-4caa-ac12-beadc5506dfe","Type":"ContainerStarted","Data":"b3aa8b3ce155b2b2e2ebcc2172ebf05c276210e7c75f822b36aa40cab8c93f2c"}
Jul 09 18:13:26.963693 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.963663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f69f701a-109d-44f3-8827-3c5c6ea7496d","Type":"ContainerStarted","Data":"da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1"}
Jul 09 18:13:26.963826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.963716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f69f701a-109d-44f3-8827-3c5c6ea7496d","Type":"ContainerStarted","Data":"a10becc0ed7144ae77ebd933985d3519872adc1c813f8aceace05ab49ce0de77"}
Jul 09 18:13:26.981469 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:26.981410    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=5.981389206 podStartE2EDuration="5.981389206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:26.979217138 +0000 UTC m=+2036.509994169" watchObservedRunningTime="2026-07-09 18:13:26.981389206 +0000 UTC m=+2036.512166237"
Jul 09 18:13:27.000919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:27.000855    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=6.000835333 podStartE2EDuration="6.000835333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:26.997825752 +0000 UTC m=+2036.528602782" watchObservedRunningTime="2026-07-09 18:13:27.000835333 +0000 UTC m=+2036.531612382"
Jul 09 18:13:27.231865 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:27.231777    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:34.163059 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:34.162992    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:13:34.185493 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:34.185449    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:13:36.231690 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:36.231647    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:13:36.233755 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:36.233713    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:13:44.163450 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:44.163398    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:13:44.185354 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:44.185302    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:13:46.231762 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:46.231717    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:13:54.163284 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:54.163240    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:13:54.185404 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:54.185359    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:13:56.231432 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:13:56.231379    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:14:04.163689 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:04.163553    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:14:04.185214 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:04.185168    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:14:06.231154 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:06.231107    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:14:14.163519 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:14.163473    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:14:14.185127 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:14.185091    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:14:16.231874 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:16.231824    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:14:24.163156 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:24.163110    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:14:24.185924 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:24.185881    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:14:26.231989 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:26.231949    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:14:31.105432 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:31.105390    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:14:31.107735 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:31.107710    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:14:34.163389 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:34.163342    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:14:34.185216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:34.185179    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:14:36.231424 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:36.231375    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:14:44.163190 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:44.163144    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:14:44.185198 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:44.185171    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:14:46.231136 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:46.231096    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:14:54.162849 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:54.162800    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:14:54.185725 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:54.185689    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:14:56.231753 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:14:56.231710    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:15:04.163704 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:04.163657    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:15:04.184775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:04.184721    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:15:06.231970 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:06.231922    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:15:14.163687 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:14.163631    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:15:14.184920 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:14.184876    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:15:16.231345 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:16.231301    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:15:24.163173 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:24.163077    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:15:24.184738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:24.184708    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:15:26.231392 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:26.231341    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:15:34.163003 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:34.162920    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:15:34.184784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:34.184749    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:15:36.231531 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:36.231479    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:15:44.162842 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:44.162795    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:15:44.185425 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:44.185391    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:15:46.231965 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:46.231925    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:15:54.163033 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:54.162987    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:15:54.184963 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:54.184927    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:15:56.231898 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:15:56.231850    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:16:04.163519 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:04.163464    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:16:04.184781 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:04.184744    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:16:06.232087 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:06.232025    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:16:14.163124 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:14.163079    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:16:14.184934 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:14.184899    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:16:16.232170 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:16.232119    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:16:24.163390 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:24.163330    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:16:24.184695 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:24.184667    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:16:26.231518 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:26.231471    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:16:34.163742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:34.163692    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:16:34.185143 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:34.185106    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:16:36.231915 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:36.231868    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:16:44.163721 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:44.163674    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": dial tcp 10.134.0.43:8001: connect: connection refused"
Jul 09 18:16:44.184719 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:44.184680    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 09 18:16:46.231786 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:46.231740    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:16:54.173498 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:54.173418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:16:54.193157 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:54.193128    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:16:54.198841 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:54.198811    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:16:54.210306 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:54.210227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:16:56.232148 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:16:56.232105    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 09 18:17:00.998631 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:00.998599    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"]
Jul 09 18:17:01.017008 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:01.016977    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"]
Jul 09 18:17:06.240983 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:06.240949    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:17:06.248173 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:06.248151    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:17:16.027965 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:16.027925    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main" containerID="cri-o://723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee" gracePeriod=45
Jul 09 18:17:16.042670 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:16.042632    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main" containerID="cri-o://55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f" gracePeriod=45
Jul 09 18:17:16.858498 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:16.858192    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="llm-d-routing-sidecar" containerID="cri-o://0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8" gracePeriod=45
Jul 09 18:17:16.991380 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:16.991354    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:17:17.098541 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098513    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tls-certs\") pod \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") "
Jul 09 18:17:17.098957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098552    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tmp-dir\") pod \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") "
Jul 09 18:17:17.098957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098571    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-model-cache\") pod \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") "
Jul 09 18:17:17.098957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098594    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-974nc\" (UniqueName: \"kubernetes.io/projected/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kube-api-access-974nc\") pod \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") "
Jul 09 18:17:17.098957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098651    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-home\") pod \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") "
Jul 09 18:17:17.098957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098672    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-dshm\") pod \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") "
Jul 09 18:17:17.098957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098729    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kserve-provision-location\") pod \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\" (UID: \"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f\") "
Jul 09 18:17:17.098957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.098873    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" (UID: "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.099354 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.099157    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.099494 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.099447    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-home" (OuterVolumeSpecName: "home") pod "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" (UID: "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.101261 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.101237    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-dshm" (OuterVolumeSpecName: "dshm") pod "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" (UID: "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.101261 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.101247    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" (UID: "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:17:17.101578 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.101549    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kube-api-access-974nc" (OuterVolumeSpecName: "kube-api-access-974nc") pod "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" (UID: "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f"). InnerVolumeSpecName "kube-api-access-974nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:17.111710 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.111680    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" (UID: "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.114831 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.114810    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:17:17.172453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.172415    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" (UID: "41e20d3a-34ab-4b17-883d-fb4dfeafaf7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.200355 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200325    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-kserve-provision-location\") pod \"1cc9b060-bcca-457a-a28d-1cb65e569db6\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") "
Jul 09 18:17:17.200505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx7hx\" (UniqueName: \"kubernetes.io/projected/1cc9b060-bcca-457a-a28d-1cb65e569db6-kube-api-access-zx7hx\") pod \"1cc9b060-bcca-457a-a28d-1cb65e569db6\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") "
Jul 09 18:17:17.200505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200402    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc9b060-bcca-457a-a28d-1cb65e569db6-tls-certs\") pod \"1cc9b060-bcca-457a-a28d-1cb65e569db6\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") "
Jul 09 18:17:17.200505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200432    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-tmp-dir\") pod \"1cc9b060-bcca-457a-a28d-1cb65e569db6\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") "
Jul 09 18:17:17.200505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200449    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-home\") pod \"1cc9b060-bcca-457a-a28d-1cb65e569db6\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") "
Jul 09 18:17:17.200505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200464    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-dshm\") pod \"1cc9b060-bcca-457a-a28d-1cb65e569db6\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") "
Jul 09 18:17:17.200782 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200539    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-model-cache\") pod \"1cc9b060-bcca-457a-a28d-1cb65e569db6\" (UID: \"1cc9b060-bcca-457a-a28d-1cb65e569db6\") "
Jul 09 18:17:17.200913 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200881    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.201002 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200921    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.201002 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200964    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.201002 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200978    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974nc\" (UniqueName: \"kubernetes.io/projected/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-kube-api-access-974nc\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.201002 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.200990    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.201229 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.201003    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.201285 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.201222    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-home" (OuterVolumeSpecName: "home") pod "1cc9b060-bcca-457a-a28d-1cb65e569db6" (UID: "1cc9b060-bcca-457a-a28d-1cb65e569db6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.201285 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.201247    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cc9b060-bcca-457a-a28d-1cb65e569db6" (UID: "1cc9b060-bcca-457a-a28d-1cb65e569db6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.202730 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.202698    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-dshm" (OuterVolumeSpecName: "dshm") pod "1cc9b060-bcca-457a-a28d-1cb65e569db6" (UID: "1cc9b060-bcca-457a-a28d-1cb65e569db6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.202951 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.202930    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc9b060-bcca-457a-a28d-1cb65e569db6-kube-api-access-zx7hx" (OuterVolumeSpecName: "kube-api-access-zx7hx") pod "1cc9b060-bcca-457a-a28d-1cb65e569db6" (UID: "1cc9b060-bcca-457a-a28d-1cb65e569db6"). InnerVolumeSpecName "kube-api-access-zx7hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:17.202994 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.202953    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc9b060-bcca-457a-a28d-1cb65e569db6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cc9b060-bcca-457a-a28d-1cb65e569db6" (UID: "1cc9b060-bcca-457a-a28d-1cb65e569db6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:17:17.213082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.213024    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cc9b060-bcca-457a-a28d-1cb65e569db6" (UID: "1cc9b060-bcca-457a-a28d-1cb65e569db6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.248952 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.248921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1cc9b060-bcca-457a-a28d-1cb65e569db6" (UID: "1cc9b060-bcca-457a-a28d-1cb65e569db6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:17.302561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.302496    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.302561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.302521    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.302561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.302532    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.302561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.302543    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.302561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.302557    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b060-bcca-457a-a28d-1cb65e569db6-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.302812 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.302571    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx7hx\" (UniqueName: \"kubernetes.io/projected/1cc9b060-bcca-457a-a28d-1cb65e569db6-kube-api-access-zx7hx\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.302812 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.302583    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc9b060-bcca-457a-a28d-1cb65e569db6-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:17:17.333766 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.333725    2576 generic.go:356] "Generic (PLEG): container finished" podID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerID="723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee" exitCode=0
Jul 09 18:17:17.333909 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.333787    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"
Jul 09 18:17:17.333909 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.333814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" event={"ID":"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f","Type":"ContainerDied","Data":"723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee"}
Jul 09 18:17:17.333909 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.333897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q" event={"ID":"41e20d3a-34ab-4b17-883d-fb4dfeafaf7f","Type":"ContainerDied","Data":"5b4fd89c25451686197bcad553d8c1308e24151f9879838c4ec3dced95efa0a2"}
Jul 09 18:17:17.334078 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.333930    2576 scope.go:117] "RemoveContainer" containerID="723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee"
Jul 09 18:17:17.336242 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.336214    2576 generic.go:356] "Generic (PLEG): container finished" podID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerID="55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f" exitCode=0
Jul 09 18:17:17.336352 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.336243    2576 generic.go:356] "Generic (PLEG): container finished" podID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerID="0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8" exitCode=0
Jul 09 18:17:17.336352 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.336279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerDied","Data":"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"}
Jul 09 18:17:17.336352 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.336290    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"
Jul 09 18:17:17.336352 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.336327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerDied","Data":"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"}
Jul 09 18:17:17.336513 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.336354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4" event={"ID":"1cc9b060-bcca-457a-a28d-1cb65e569db6","Type":"ContainerDied","Data":"df9becc0564ca9af6d7be9ce48fb76a32642987ae80682d6b3e87b590437b6b2"}
Jul 09 18:17:17.346026 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.346008    2576 scope.go:117] "RemoveContainer" containerID="191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d"
Jul 09 18:17:17.355576 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.355555    2576 scope.go:117] "RemoveContainer" containerID="723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee"
Jul 09 18:17:17.355811 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:17:17.355796    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee\": container with ID starting with 723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee not found: ID does not exist" containerID="723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee"
Jul 09 18:17:17.355865 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.355818    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee"} err="failed to get container status \"723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee\": rpc error: code = NotFound desc = could not find container \"723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee\": container with ID starting with 723e4fcdc0ba09e599c11efc71963ba00568a3698e8fbd09079da12bf7980aee not found: ID does not exist"
Jul 09 18:17:17.355865 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.355834    2576 scope.go:117] "RemoveContainer" containerID="191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d"
Jul 09 18:17:17.356090 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:17:17.356074    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d\": container with ID starting with 191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d not found: ID does not exist" containerID="191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d"
Jul 09 18:17:17.356141 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.356097    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d"} err="failed to get container status \"191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d\": rpc error: code = NotFound desc = could not find container \"191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d\": container with ID starting with 191bbdb3f968644547d3b4fc49ca6bd4e58b2405e801db75c0ace74f7071869d not found: ID does not exist"
Jul 09 18:17:17.356141 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.356112    2576 scope.go:117] "RemoveContainer" containerID="55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"
Jul 09 18:17:17.362867 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.362799    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"]
Jul 09 18:17:17.367497 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.367477    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7948w2q"]
Jul 09 18:17:17.368052 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.368020    2576 scope.go:117] "RemoveContainer" containerID="a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"
Jul 09 18:17:17.377813 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.377168    2576 scope.go:117] "RemoveContainer" containerID="0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"
Jul 09 18:17:17.381203 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.381182    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"]
Jul 09 18:17:17.385430 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.385408    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5cc9f5567d5bzm4"]
Jul 09 18:17:17.386849 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.386830    2576 scope.go:117] "RemoveContainer" containerID="55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"
Jul 09 18:17:17.387139 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:17:17.387108    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f\": container with ID starting with 55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f not found: ID does not exist" containerID="55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"
Jul 09 18:17:17.387242 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387136    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"} err="failed to get container status \"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f\": rpc error: code = NotFound desc = could not find container \"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f\": container with ID starting with 55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f not found: ID does not exist"
Jul 09 18:17:17.387242 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387153    2576 scope.go:117] "RemoveContainer" containerID="a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"
Jul 09 18:17:17.387410 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:17:17.387392    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3\": container with ID starting with a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3 not found: ID does not exist" containerID="a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"
Jul 09 18:17:17.387475 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387417    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"} err="failed to get container status \"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3\": rpc error: code = NotFound desc = could not find container \"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3\": container with ID starting with a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3 not found: ID does not exist"
Jul 09 18:17:17.387475 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387435    2576 scope.go:117] "RemoveContainer" containerID="0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"
Jul 09 18:17:17.387688 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:17:17.387670    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8\": container with ID starting with 0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8 not found: ID does not exist" containerID="0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"
Jul 09 18:17:17.387742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387692    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"} err="failed to get container status \"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8\": rpc error: code = NotFound desc = could not find container \"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8\": container with ID starting with 0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8 not found: ID does not exist"
Jul 09 18:17:17.387742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387704    2576 scope.go:117] "RemoveContainer" containerID="55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"
Jul 09 18:17:17.387955 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387936    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f"} err="failed to get container status \"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f\": rpc error: code = NotFound desc = could not find container \"55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f\": container with ID starting with 55e8978f20f896f48733dddbb95f05de98bc5c2ade10ac48f9371c6bf43dc29f not found: ID does not exist"
Jul 09 18:17:17.388015 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.387955    2576 scope.go:117] "RemoveContainer" containerID="a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"
Jul 09 18:17:17.388183 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.388165    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3"} err="failed to get container status \"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3\": rpc error: code = NotFound desc = could not find container \"a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3\": container with ID starting with a69e63b12a6954e1f3000d13e6a6003681ae9412db929c4b4a03122d886842f3 not found: ID does not exist"
Jul 09 18:17:17.388246 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.388185    2576 scope.go:117] "RemoveContainer" containerID="0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"
Jul 09 18:17:17.388376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.388360    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8"} err="failed to get container status \"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8\": rpc error: code = NotFound desc = could not find container \"0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8\": container with ID starting with 0b859f34a0eb13980ff07d609a49eabdb8867c18c3be43978e3f5c18e951bad8 not found: ID does not exist"
Jul 09 18:17:17.670704 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.670673    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" path="/var/lib/kubelet/pods/1cc9b060-bcca-457a-a28d-1cb65e569db6/volumes"
Jul 09 18:17:17.671162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:17.671148    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" path="/var/lib/kubelet/pods/41e20d3a-34ab-4b17-883d-fb4dfeafaf7f/volumes"
Jul 09 18:17:30.743427 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743392    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"]
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743779    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="llm-d-routing-sidecar"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743791    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="llm-d-routing-sidecar"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743809    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="storage-initializer"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743814    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="storage-initializer"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743824    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="storage-initializer"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743830    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="storage-initializer"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743836    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743841    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743847    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main"
Jul 09 18:17:30.743895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743852    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main"
Jul 09 18:17:30.744235 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743908    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="llm-d-routing-sidecar"
Jul 09 18:17:30.744235 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743920    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e20d3a-34ab-4b17-883d-fb4dfeafaf7f" containerName="main"
Jul 09 18:17:30.744235 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.743926    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc9b060-bcca-457a-a28d-1cb65e569db6" containerName="main"
Jul 09 18:17:30.747805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.747782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.750760 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.750740    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-cpht4\""
Jul 09 18:17:30.750868 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.750742    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 18:17:30.759460 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.759367    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"]
Jul 09 18:17:30.762396 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.762372    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"]
Jul 09 18:17:30.765786 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.765767    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.778976 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.778956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"]
Jul 09 18:17:30.840216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.840216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.840429 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-dshm\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.840429 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.840429 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.840429 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25ldl\" (UniqueName: \"kubernetes.io/projected/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kube-api-access-25ldl\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.840603 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtct6\" (UniqueName: \"kubernetes.io/projected/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kube-api-access-qtct6\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.840603 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.840603 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.840603 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-model-cache\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.840603 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.840806 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.840806 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.840806 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.840767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-home\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.941440 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.941619 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.941619 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25ldl\" (UniqueName: \"kubernetes.io/projected/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kube-api-access-25ldl\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.941619 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtct6\" (UniqueName: \"kubernetes.io/projected/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kube-api-access-qtct6\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.941619 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.941619 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.941875 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-model-cache\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.941875 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.941875 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.941875 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.941875 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-home\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.941875 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941837    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.941875 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.942299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.942299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.941922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-dshm\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.942299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.942026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.942299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.942075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-model-cache\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.942299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.942132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-home\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.942299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.942177    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.942299 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.942214    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.942614 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.942347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.942614 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.942435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.943858 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.943840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.944093 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.944076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-dshm\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.944453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.944436    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 18:17:30.952025 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.952005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtct6\" (UniqueName: \"kubernetes.io/projected/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kube-api-access-qtct6\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:30.952870 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.952843    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25ldl\" (UniqueName: \"kubernetes.io/projected/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kube-api-access-25ldl\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.954410 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.954386    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:30.954572 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:30.954545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-68796975c7-ddffs\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:31.061306 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.061282    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-cpht4\""
Jul 09 18:17:31.069452 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.069436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:31.078332 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.078302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:31.230865 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.230839    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"]
Jul 09 18:17:31.232610 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:17:31.232579    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7948b0_afb2_4d45_9e65_bd7dc3175ab6.slice/crio-2507b258bcb186404d9e3ec33123032dcb774abefa90a78d1598c1a3d2ac164a WatchSource:0}: Error finding container 2507b258bcb186404d9e3ec33123032dcb774abefa90a78d1598c1a3d2ac164a: Status 404 returned error can't find the container with id 2507b258bcb186404d9e3ec33123032dcb774abefa90a78d1598c1a3d2ac164a
Jul 09 18:17:31.412489 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.412384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerStarted","Data":"334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec"}
Jul 09 18:17:31.412489 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.412456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerStarted","Data":"2507b258bcb186404d9e3ec33123032dcb774abefa90a78d1598c1a3d2ac164a"}
Jul 09 18:17:31.412708 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.412574    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:31.457222 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:31.457189    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"]
Jul 09 18:17:31.458778 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:17:31.458753    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec4c7e9_9050_4477_bbd0_843e5fa80cbd.slice/crio-56c2d6f9c64cc51722c6ff062c8cdab41677e70dfa7b2976baabb0d93b2c8c65 WatchSource:0}: Error finding container 56c2d6f9c64cc51722c6ff062c8cdab41677e70dfa7b2976baabb0d93b2c8c65: Status 404 returned error can't find the container with id 56c2d6f9c64cc51722c6ff062c8cdab41677e70dfa7b2976baabb0d93b2c8c65
Jul 09 18:17:32.419394 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:32.419358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerStarted","Data":"bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1"}
Jul 09 18:17:32.421095 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:32.421060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" event={"ID":"dec4c7e9-9050-4477-bbd0-843e5fa80cbd","Type":"ContainerStarted","Data":"7b4bbe9dc4ae9ef7754e04d394d6e900bbdeb80faf219d798a79366c52382a74"}
Jul 09 18:17:32.421204 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:32.421114    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" event={"ID":"dec4c7e9-9050-4477-bbd0-843e5fa80cbd","Type":"ContainerStarted","Data":"56c2d6f9c64cc51722c6ff062c8cdab41677e70dfa7b2976baabb0d93b2c8c65"}
Jul 09 18:17:35.448362 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:35.448278    2576 generic.go:356] "Generic (PLEG): container finished" podID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerID="bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1" exitCode=0
Jul 09 18:17:35.448756 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:35.448355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerDied","Data":"bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1"}
Jul 09 18:17:35.449954 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:35.449930    2576 generic.go:356] "Generic (PLEG): container finished" podID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerID="7b4bbe9dc4ae9ef7754e04d394d6e900bbdeb80faf219d798a79366c52382a74" exitCode=0
Jul 09 18:17:35.450076 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:35.449993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" event={"ID":"dec4c7e9-9050-4477-bbd0-843e5fa80cbd","Type":"ContainerDied","Data":"7b4bbe9dc4ae9ef7754e04d394d6e900bbdeb80faf219d798a79366c52382a74"}
Jul 09 18:17:36.458567 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:36.458525    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerStarted","Data":"3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae"}
Jul 09 18:17:36.460445 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:36.460419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" event={"ID":"dec4c7e9-9050-4477-bbd0-843e5fa80cbd","Type":"ContainerStarted","Data":"08673b9a514124de5782e0c4da4e7e63373ccd4a0b55c9e7d0e4cb3fbc8aca8f"}
Jul 09 18:17:36.487382 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:36.487321    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podStartSLOduration=2.813092451 podStartE2EDuration="6.487300354s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.674207903s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:36.483975544 +0000 UTC m=+2286.014752599" watchObservedRunningTime="2026-07-09 18:17:36.487300354 +0000 UTC m=+2286.018077384"
Jul 09 18:17:36.508311 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:36.508260    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podStartSLOduration=3.139740835 podStartE2EDuration="6.508245391s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.368504556s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:36.506215519 +0000 UTC m=+2286.036992550" watchObservedRunningTime="2026-07-09 18:17:36.508245391 +0000 UTC m=+2286.039022420"
Jul 09 18:17:41.070219 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:41.070173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:41.070707 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:41.070299    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:41.071572 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:41.071534    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:17:41.078497 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:41.078427    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:41.078497 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:41.078489    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:17:41.079687 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:41.079660    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:17:41.086242 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:41.086215    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:17:51.070148 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:51.070108    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:17:51.079175 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:51.079133    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:17:59.343884 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:17:59.343836    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:18:01.070778 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:01.070731    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:18:01.079529 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:01.079491    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:18:11.070462 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:11.070418    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:18:11.079162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:11.079124    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:18:14.372895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:14.372821    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main" containerID="cri-o://da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1" gracePeriod=45
Jul 09 18:18:15.150105 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.150081    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:18:15.174010 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.173954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-model-cache\") pod \"f69f701a-109d-44f3-8827-3c5c6ea7496d\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") "
Jul 09 18:18:15.174010 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.173999    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-home\") pod \"f69f701a-109d-44f3-8827-3c5c6ea7496d\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") "
Jul 09 18:18:15.174281 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.174033    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-tmp-dir\") pod \"f69f701a-109d-44f3-8827-3c5c6ea7496d\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") "
Jul 09 18:18:15.174281 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.174176    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"f69f701a-109d-44f3-8827-3c5c6ea7496d\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") "
Jul 09 18:18:15.174281 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.174270    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6crz\" (UniqueName: \"kubernetes.io/projected/f69f701a-109d-44f3-8827-3c5c6ea7496d-kube-api-access-j6crz\") pod \"f69f701a-109d-44f3-8827-3c5c6ea7496d\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") "
Jul 09 18:18:15.174456 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.174305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-dshm\") pod \"f69f701a-109d-44f3-8827-3c5c6ea7496d\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") "
Jul 09 18:18:15.174456 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.174327    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69f701a-109d-44f3-8827-3c5c6ea7496d-tls-certs\") pod \"f69f701a-109d-44f3-8827-3c5c6ea7496d\" (UID: \"f69f701a-109d-44f3-8827-3c5c6ea7496d\") "
Jul 09 18:18:15.175747 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.175487    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f69f701a-109d-44f3-8827-3c5c6ea7496d" (UID: "f69f701a-109d-44f3-8827-3c5c6ea7496d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:15.177375 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.177335    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-home" (OuterVolumeSpecName: "home") pod "f69f701a-109d-44f3-8827-3c5c6ea7496d" (UID: "f69f701a-109d-44f3-8827-3c5c6ea7496d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:15.179688 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.179616    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-dshm" (OuterVolumeSpecName: "dshm") pod "f69f701a-109d-44f3-8827-3c5c6ea7496d" (UID: "f69f701a-109d-44f3-8827-3c5c6ea7496d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:15.179688 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.179659    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69f701a-109d-44f3-8827-3c5c6ea7496d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f69f701a-109d-44f3-8827-3c5c6ea7496d" (UID: "f69f701a-109d-44f3-8827-3c5c6ea7496d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:15.180058 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.180014    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69f701a-109d-44f3-8827-3c5c6ea7496d-kube-api-access-j6crz" (OuterVolumeSpecName: "kube-api-access-j6crz") pod "f69f701a-109d-44f3-8827-3c5c6ea7496d" (UID: "f69f701a-109d-44f3-8827-3c5c6ea7496d"). InnerVolumeSpecName "kube-api-access-j6crz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:15.184612 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.184589    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf" (OuterVolumeSpecName: "kserve-pvc-source") pod "f69f701a-109d-44f3-8827-3c5c6ea7496d" (UID: "f69f701a-109d-44f3-8827-3c5c6ea7496d"). InnerVolumeSpecName "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:18:15.195407 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.195376    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f69f701a-109d-44f3-8827-3c5c6ea7496d" (UID: "f69f701a-109d-44f3-8827-3c5c6ea7496d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:15.275553 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.275514    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:15.275553 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.275544    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:15.275553 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.275556    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:15.275784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.275568    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6crz\" (UniqueName: \"kubernetes.io/projected/f69f701a-109d-44f3-8827-3c5c6ea7496d-kube-api-access-j6crz\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:15.275784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.275582    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69f701a-109d-44f3-8827-3c5c6ea7496d-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:15.275784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.275593    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69f701a-109d-44f3-8827-3c5c6ea7496d-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:15.714224 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.714189    2576 generic.go:356] "Generic (PLEG): container finished" podID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerID="da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1" exitCode=0
Jul 09 18:18:15.714677 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.714257    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:18:15.714677 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.714279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f69f701a-109d-44f3-8827-3c5c6ea7496d","Type":"ContainerDied","Data":"da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1"}
Jul 09 18:18:15.714677 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.714336    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"f69f701a-109d-44f3-8827-3c5c6ea7496d","Type":"ContainerDied","Data":"a10becc0ed7144ae77ebd933985d3519872adc1c813f8aceace05ab49ce0de77"}
Jul 09 18:18:15.714677 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.714365    2576 scope.go:117] "RemoveContainer" containerID="da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1"
Jul 09 18:18:15.724542 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.724516    2576 scope.go:117] "RemoveContainer" containerID="da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1"
Jul 09 18:18:15.724863 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:18:15.724835    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1\": container with ID starting with da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1 not found: ID does not exist" containerID="da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1"
Jul 09 18:18:15.724943 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.724869    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1"} err="failed to get container status \"da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1\": rpc error: code = NotFound desc = could not find container \"da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1\": container with ID starting with da931c0a714916edf5811f1e3b38e50748b6722aae4d9e1add2fbfa7586df9a1 not found: ID does not exist"
Jul 09 18:18:15.738464 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.738433    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:18:15.740420 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.740397    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:18:15.784345 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:15.784313    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:18:17.670617 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:17.670574    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" path="/var/lib/kubelet/pods/f69f701a-109d-44f3-8827-3c5c6ea7496d/volumes"
Jul 09 18:18:21.070351 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:21.070306    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:18:21.078737 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:21.078701    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:18:30.808498 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:30.808376    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="ffd15146-1c07-4caa-ac12-beadc5506dfe" containerName="main" containerID="cri-o://7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31" gracePeriod=45
Jul 09 18:18:31.070478 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.070417    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:18:31.079519 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.079470    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:18:31.573186 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.573156    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:18:31.646273 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646168    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-home\") pod \"ffd15146-1c07-4caa-ac12-beadc5506dfe\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") "
Jul 09 18:18:31.646273 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646212    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-tmp-dir\") pod \"ffd15146-1c07-4caa-ac12-beadc5506dfe\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") "
Jul 09 18:18:31.646273 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646241    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd15146-1c07-4caa-ac12-beadc5506dfe-tls-certs\") pod \"ffd15146-1c07-4caa-ac12-beadc5506dfe\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") "
Jul 09 18:18:31.646554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646287    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-model-cache\") pod \"ffd15146-1c07-4caa-ac12-beadc5506dfe\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") "
Jul 09 18:18:31.646554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"ffd15146-1c07-4caa-ac12-beadc5506dfe\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") "
Jul 09 18:18:31.646554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dq7c\" (UniqueName: \"kubernetes.io/projected/ffd15146-1c07-4caa-ac12-beadc5506dfe-kube-api-access-2dq7c\") pod \"ffd15146-1c07-4caa-ac12-beadc5506dfe\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") "
Jul 09 18:18:31.646554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646508    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-dshm\") pod \"ffd15146-1c07-4caa-ac12-beadc5506dfe\" (UID: \"ffd15146-1c07-4caa-ac12-beadc5506dfe\") "
Jul 09 18:18:31.646758 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646598    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffd15146-1c07-4caa-ac12-beadc5506dfe" (UID: "ffd15146-1c07-4caa-ac12-beadc5506dfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:31.647178 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646881    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:31.647178 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.646916    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-home" (OuterVolumeSpecName: "home") pod "ffd15146-1c07-4caa-ac12-beadc5506dfe" (UID: "ffd15146-1c07-4caa-ac12-beadc5506dfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:31.649220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.649160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd15146-1c07-4caa-ac12-beadc5506dfe-kube-api-access-2dq7c" (OuterVolumeSpecName: "kube-api-access-2dq7c") pod "ffd15146-1c07-4caa-ac12-beadc5506dfe" (UID: "ffd15146-1c07-4caa-ac12-beadc5506dfe"). InnerVolumeSpecName "kube-api-access-2dq7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:31.649349 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.649252    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-dshm" (OuterVolumeSpecName: "dshm") pod "ffd15146-1c07-4caa-ac12-beadc5506dfe" (UID: "ffd15146-1c07-4caa-ac12-beadc5506dfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:31.649476 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.649455    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd15146-1c07-4caa-ac12-beadc5506dfe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffd15146-1c07-4caa-ac12-beadc5506dfe" (UID: "ffd15146-1c07-4caa-ac12-beadc5506dfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:31.650029 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.650006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf" (OuterVolumeSpecName: "kserve-pvc-source") pod "ffd15146-1c07-4caa-ac12-beadc5506dfe" (UID: "ffd15146-1c07-4caa-ac12-beadc5506dfe"). InnerVolumeSpecName "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:18:31.667120 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.667082    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffd15146-1c07-4caa-ac12-beadc5506dfe" (UID: "ffd15146-1c07-4caa-ac12-beadc5506dfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:31.748234 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.748192    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:31.748234 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.748229    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:31.748234 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.748242    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd15146-1c07-4caa-ac12-beadc5506dfe-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:31.748488 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.748282    2576 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-132-16.ec2.internal\" "
Jul 09 18:18:31.748488 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.748298    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dq7c\" (UniqueName: \"kubernetes.io/projected/ffd15146-1c07-4caa-ac12-beadc5506dfe-kube-api-access-2dq7c\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:31.748488 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.748311    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd15146-1c07-4caa-ac12-beadc5506dfe-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:18:31.788692 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.788666    2576 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf") on node "ip-10-0-132-16.ec2.internal"
Jul 09 18:18:31.813752 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.813723    2576 generic.go:356] "Generic (PLEG): container finished" podID="ffd15146-1c07-4caa-ac12-beadc5506dfe" containerID="7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31" exitCode=0
Jul 09 18:18:31.814161 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.813779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ffd15146-1c07-4caa-ac12-beadc5506dfe","Type":"ContainerDied","Data":"7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31"}
Jul 09 18:18:31.814161 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.813793    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:18:31.814161 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.813822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"ffd15146-1c07-4caa-ac12-beadc5506dfe","Type":"ContainerDied","Data":"b3aa8b3ce155b2b2e2ebcc2172ebf05c276210e7c75f822b36aa40cab8c93f2c"}
Jul 09 18:18:31.814161 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.813846    2576 scope.go:117] "RemoveContainer" containerID="7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31"
Jul 09 18:18:31.823500 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.823414    2576 scope.go:117] "RemoveContainer" containerID="7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31"
Jul 09 18:18:31.823785 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:18:31.823758    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31\": container with ID starting with 7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31 not found: ID does not exist" containerID="7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31"
Jul 09 18:18:31.823913 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.823793    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31"} err="failed to get container status \"7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31\": rpc error: code = NotFound desc = could not find container \"7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31\": container with ID starting with 7990320a4e6c867a51c0e7917c35fb59d6f95ada52794fc41ba6f5dcfcd9de31 not found: ID does not exist"
Jul 09 18:18:31.832937 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.832909    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:18:31.836213 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.836191    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:18:31.849188 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:31.849160    2576 reconciler_common.go:299] "Volume detached for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"csi-4598e405477c941d84bc8ea9595d70de15d8d82f94fda290233f9c0e59ad69ca\""
Jul 09 18:18:33.671178 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:33.671140    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd15146-1c07-4caa-ac12-beadc5506dfe" path="/var/lib/kubelet/pods/ffd15146-1c07-4caa-ac12-beadc5506dfe/volumes"
Jul 09 18:18:36.810627 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.810594    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"]
Jul 09 18:18:36.811029 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.810990    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd15146-1c07-4caa-ac12-beadc5506dfe" containerName="main"
Jul 09 18:18:36.811029 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.811001    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd15146-1c07-4caa-ac12-beadc5506dfe" containerName="main"
Jul 09 18:18:36.811029 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.811017    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main"
Jul 09 18:18:36.811029 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.811023    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main"
Jul 09 18:18:36.811214 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.811106    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd15146-1c07-4caa-ac12-beadc5506dfe" containerName="main"
Jul 09 18:18:36.811214 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.811115    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f69f701a-109d-44f3-8827-3c5c6ea7496d" containerName="main"
Jul 09 18:18:36.813363 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.813342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.820977 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:18:36.820541    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"stop-feature-test-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-132-16.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-132-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\"" type="*v1.Secret"
Jul 09 18:18:36.820977 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:18:36.820607    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"stop-feature-test-kserve-745fc78d48-m5r6h\" is forbidden: User \"system:node:ip-10-0-132-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-132-16.ec2.internal' and this object" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.873092 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.873021    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"]
Jul 09 18:18:36.895921 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.895881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-home\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.896115 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.895940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlkz\" (UniqueName: \"kubernetes.io/projected/629d5b8e-1118-4d76-9ade-835a12406a64-kube-api-access-8nlkz\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.896115 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.896020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.896214 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.896132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-kserve-provision-location\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.896253 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.896228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-model-cache\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.896305 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.896266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-dshm\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.896360 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.896337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-tmp-dir\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997195 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-kserve-provision-location\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997361 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-model-cache\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997361 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-dshm\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997361 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-tmp-dir\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997559 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-home\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997559 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlkz\" (UniqueName: \"kubernetes.io/projected/629d5b8e-1118-4d76-9ade-835a12406a64-kube-api-access-8nlkz\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997559 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-kserve-provision-location\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997663    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-model-cache\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997859 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-tmp-dir\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.997910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.997865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-home\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:36.999651 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:36.999625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-dshm\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:37.007688 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:37.007657    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlkz\" (UniqueName: \"kubernetes.io/projected/629d5b8e-1118-4d76-9ade-835a12406a64-kube-api-access-8nlkz\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:37.998309 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:18:37.998272    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Jul 09 18:18:37.998753 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:18:37.998361    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs podName:629d5b8e-1118-4d76-9ade-835a12406a64 nodeName:}" failed. No retries permitted until 2026-07-09 18:18:38.498342833 +0000 UTC m=+2348.029119841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs") pod "stop-feature-test-kserve-745fc78d48-m5r6h" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64") : failed to sync secret cache: timed out waiting for the condition
Jul 09 18:18:38.184255 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:38.184227    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 18:18:38.513095 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:38.513057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:38.515838 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:38.515810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs\") pod \"stop-feature-test-kserve-745fc78d48-m5r6h\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:38.625151 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:38.625083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:38.785837 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:38.785802    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"]
Jul 09 18:18:38.787105 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:18:38.787067    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629d5b8e_1118_4d76_9ade_835a12406a64.slice/crio-2a4edb8f41d10d584033d036dd67c00ea2e8d6763a9b6df0bf08883c2a2afc67 WatchSource:0}: Error finding container 2a4edb8f41d10d584033d036dd67c00ea2e8d6763a9b6df0bf08883c2a2afc67: Status 404 returned error can't find the container with id 2a4edb8f41d10d584033d036dd67c00ea2e8d6763a9b6df0bf08883c2a2afc67
Jul 09 18:18:38.860223 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:38.860187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" event={"ID":"629d5b8e-1118-4d76-9ade-835a12406a64","Type":"ContainerStarted","Data":"48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf"}
Jul 09 18:18:38.860382 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:38.860258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" event={"ID":"629d5b8e-1118-4d76-9ade-835a12406a64","Type":"ContainerStarted","Data":"2a4edb8f41d10d584033d036dd67c00ea2e8d6763a9b6df0bf08883c2a2afc67"}
Jul 09 18:18:41.070114 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:41.070058    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:18:41.079678 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:41.079631    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:18:42.896281 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:42.896245    2576 generic.go:356] "Generic (PLEG): container finished" podID="629d5b8e-1118-4d76-9ade-835a12406a64" containerID="48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf" exitCode=0
Jul 09 18:18:42.896779 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:42.896319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" event={"ID":"629d5b8e-1118-4d76-9ade-835a12406a64","Type":"ContainerDied","Data":"48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf"}
Jul 09 18:18:43.904673 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:43.904639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" event={"ID":"629d5b8e-1118-4d76-9ade-835a12406a64","Type":"ContainerStarted","Data":"9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e"}
Jul 09 18:18:43.931520 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:43.931431    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podStartSLOduration=4.4759583339999995 podStartE2EDuration="7.93141208s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.455453746s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:43.928784259 +0000 UTC m=+2353.459561322" watchObservedRunningTime="2026-07-09 18:18:43.93141208 +0000 UTC m=+2353.462189489"
Jul 09 18:18:44.625395 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:44.625351    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:48.626077 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:48.625234    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:18:48.626802 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:48.626763    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:18:51.070633 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:51.070574    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:18:51.079458 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:51.079409    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:18:58.625568 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:18:58.625520    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:19:01.070247 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:01.070197    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:19:01.079336 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:01.079294    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:19:08.625536 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:08.625489    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:19:11.070643 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:11.070596    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:19:11.079122 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:11.079075    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:19:18.625562 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:18.625501    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:19:21.070465 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:21.070414    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:19:21.078633 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:21.078596    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:19:28.626406 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:28.626364    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:19:31.070680 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:31.070617    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:19:31.079147 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:31.079104    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:19:31.143127 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:31.143098    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:19:31.145771 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:31.145752    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:19:38.625595 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:38.625549    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:19:41.070421 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:41.070359    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:19:41.078625 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:41.078588    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:19:48.625688 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:48.625642    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:19:51.070683 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:51.070633    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:19:51.079013 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:51.078973    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:19:58.625587 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:19:58.625483    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:20:01.070710 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:01.070658    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:20:01.078668 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:01.078623    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:20:08.625451 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:08.625412    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:20:11.069855 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:11.069810    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8001/health\": dial tcp 10.134.0.47:8001: connect: connection refused"
Jul 09 18:20:11.078624 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:11.078590    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 09 18:20:18.625628 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:18.625583    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:20:21.079659 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:21.079629    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:20:21.092569 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:21.092541    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:20:21.097883 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:21.097860    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:20:21.102955 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:21.102931    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:20:28.626143 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:28.626090    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:20:30.529354 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:30.529316    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"]
Jul 09 18:20:30.536791 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:30.536754    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"]
Jul 09 18:20:38.626493 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:38.626454    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:20:45.554421 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:45.554368    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main" containerID="cri-o://3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae" gracePeriod=45
Jul 09 18:20:45.562125 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:45.562090    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main" containerID="cri-o://08673b9a514124de5782e0c4da4e7e63373ccd4a0b55c9e7d0e4cb3fbc8aca8f" gracePeriod=45
Jul 09 18:20:46.570644 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.570513    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="llm-d-routing-sidecar" containerID="cri-o://334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec" gracePeriod=44
Jul 09 18:20:46.605324 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.605296    2576 generic.go:356] "Generic (PLEG): container finished" podID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerID="3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae" exitCode=0
Jul 09 18:20:46.605445 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.605363    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerDied","Data":"3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae"}
Jul 09 18:20:46.607461 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.607442    2576 generic.go:356] "Generic (PLEG): container finished" podID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerID="08673b9a514124de5782e0c4da4e7e63373ccd4a0b55c9e7d0e4cb3fbc8aca8f" exitCode=0
Jul 09 18:20:46.607557 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.607516    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" event={"ID":"dec4c7e9-9050-4477-bbd0-843e5fa80cbd","Type":"ContainerDied","Data":"08673b9a514124de5782e0c4da4e7e63373ccd4a0b55c9e7d0e4cb3fbc8aca8f"}
Jul 09 18:20:46.724226 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.724203    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:20:46.836182 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836155    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:20:46.836473 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836453    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-home\") pod \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") "
Jul 09 18:20:46.836574 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836497    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25ldl\" (UniqueName: \"kubernetes.io/projected/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kube-api-access-25ldl\") pod \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") "
Jul 09 18:20:46.836574 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836526    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kserve-provision-location\") pod \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") "
Jul 09 18:20:46.836574 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836562    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-model-cache\") pod \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") "
Jul 09 18:20:46.836734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836623    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tls-certs\") pod \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") "
Jul 09 18:20:46.836734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836638    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tmp-dir\") pod \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") "
Jul 09 18:20:46.836734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-dshm\") pod \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\" (UID: \"dec4c7e9-9050-4477-bbd0-843e5fa80cbd\") "
Jul 09 18:20:46.837066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.836996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "dec4c7e9-9050-4477-bbd0-843e5fa80cbd" (UID: "dec4c7e9-9050-4477-bbd0-843e5fa80cbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.837179 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.837134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-home" (OuterVolumeSpecName: "home") pod "dec4c7e9-9050-4477-bbd0-843e5fa80cbd" (UID: "dec4c7e9-9050-4477-bbd0-843e5fa80cbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.838733 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.838705    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kube-api-access-25ldl" (OuterVolumeSpecName: "kube-api-access-25ldl") pod "dec4c7e9-9050-4477-bbd0-843e5fa80cbd" (UID: "dec4c7e9-9050-4477-bbd0-843e5fa80cbd"). InnerVolumeSpecName "kube-api-access-25ldl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:46.839278 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.839245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-dshm" (OuterVolumeSpecName: "dshm") pod "dec4c7e9-9050-4477-bbd0-843e5fa80cbd" (UID: "dec4c7e9-9050-4477-bbd0-843e5fa80cbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.839278 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.839265    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dec4c7e9-9050-4477-bbd0-843e5fa80cbd" (UID: "dec4c7e9-9050-4477-bbd0-843e5fa80cbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:46.854350 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.854319    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dec4c7e9-9050-4477-bbd0-843e5fa80cbd" (UID: "dec4c7e9-9050-4477-bbd0-843e5fa80cbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.912418 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.912365    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dec4c7e9-9050-4477-bbd0-843e5fa80cbd" (UID: "dec4c7e9-9050-4477-bbd0-843e5fa80cbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.938180 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938156    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tmp-dir\") pod \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") "
Jul 09 18:20:46.938338 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938189    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kserve-provision-location\") pod \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") "
Jul 09 18:20:46.938338 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938225    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-home\") pod \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") "
Jul 09 18:20:46.938338 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938261    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tls-certs\") pod \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") "
Jul 09 18:20:46.938338 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938319    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-dshm\") pod \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") "
Jul 09 18:20:46.938632 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-model-cache\") pod \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") "
Jul 09 18:20:46.938632 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938383    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtct6\" (UniqueName: \"kubernetes.io/projected/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kube-api-access-qtct6\") pod \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\" (UID: \"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6\") "
Jul 09 18:20:46.938753 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938739    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:46.938813 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938757    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:46.938813 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938770    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:46.938813 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938782    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:46.938813 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938795    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25ldl\" (UniqueName: \"kubernetes.io/projected/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kube-api-access-25ldl\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:46.938813 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938808    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:46.939119 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938822    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec4c7e9-9050-4477-bbd0-843e5fa80cbd-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:46.939119 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.938872    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-home" (OuterVolumeSpecName: "home") pod "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" (UID: "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.939119 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.939053    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" (UID: "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.941151 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.941124    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" (UID: "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:46.941272 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.941255    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-dshm" (OuterVolumeSpecName: "dshm") pod "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" (UID: "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:46.941356 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.941334    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kube-api-access-qtct6" (OuterVolumeSpecName: "kube-api-access-qtct6") pod "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" (UID: "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6"). InnerVolumeSpecName "kube-api-access-qtct6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:46.950572 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:46.950550    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" (UID: "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:47.016438 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.016387    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" (UID: "1b7948b0-afb2-4d45-9e65-bd7dc3175ab6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:47.039444 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.039414    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:47.039444 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.039439    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:47.039444 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.039449    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:47.039646 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.039457    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:47.039646 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.039465    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtct6\" (UniqueName: \"kubernetes.io/projected/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kube-api-access-qtct6\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:47.039646 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.039473    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:47.039646 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.039482    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:20:47.614469 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.614437    2576 generic.go:356] "Generic (PLEG): container finished" podID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerID="334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec" exitCode=0
Jul 09 18:20:47.614895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.614517    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"
Jul 09 18:20:47.614895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.614525    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerDied","Data":"334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec"}
Jul 09 18:20:47.614895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.614579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs" event={"ID":"1b7948b0-afb2-4d45-9e65-bd7dc3175ab6","Type":"ContainerDied","Data":"2507b258bcb186404d9e3ec33123032dcb774abefa90a78d1598c1a3d2ac164a"}
Jul 09 18:20:47.614895 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.614597    2576 scope.go:117] "RemoveContainer" containerID="3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae"
Jul 09 18:20:47.616319 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.616300    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"
Jul 09 18:20:47.616474 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.616314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw" event={"ID":"dec4c7e9-9050-4477-bbd0-843e5fa80cbd","Type":"ContainerDied","Data":"56c2d6f9c64cc51722c6ff062c8cdab41677e70dfa7b2976baabb0d93b2c8c65"}
Jul 09 18:20:47.627374 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.627347    2576 scope.go:117] "RemoveContainer" containerID="bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1"
Jul 09 18:20:47.642084 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.642072    2576 scope.go:117] "RemoveContainer" containerID="334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec"
Jul 09 18:20:47.650919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.650898    2576 scope.go:117] "RemoveContainer" containerID="3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae"
Jul 09 18:20:47.651162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.651142    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"]
Jul 09 18:20:47.651242 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:20:47.651219    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae\": container with ID starting with 3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae not found: ID does not exist" containerID="3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae"
Jul 09 18:20:47.651311 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.651253    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae"} err="failed to get container status \"3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae\": rpc error: code = NotFound desc = could not find container \"3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae\": container with ID starting with 3d11967536c21aeb3826e6ece63737a8697efb586d4d8aa019ec94f71a8e6eae not found: ID does not exist"
Jul 09 18:20:47.651311 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.651278    2576 scope.go:117] "RemoveContainer" containerID="bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1"
Jul 09 18:20:47.651550 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:20:47.651533    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1\": container with ID starting with bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1 not found: ID does not exist" containerID="bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1"
Jul 09 18:20:47.651591 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.651555    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1"} err="failed to get container status \"bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1\": rpc error: code = NotFound desc = could not find container \"bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1\": container with ID starting with bb913a6ebb5e41d07656e263834b08b15cf4f236e15d2147365fd5644ee03dc1 not found: ID does not exist"
Jul 09 18:20:47.651591 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.651570    2576 scope.go:117] "RemoveContainer" containerID="334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec"
Jul 09 18:20:47.651849 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:20:47.651830    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec\": container with ID starting with 334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec not found: ID does not exist" containerID="334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec"
Jul 09 18:20:47.651907 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.651853    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec"} err="failed to get container status \"334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec\": rpc error: code = NotFound desc = could not find container \"334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec\": container with ID starting with 334a5d4cddcc9f039d5458120caeb14f72ef68a7d868cbf9ad1dbc51f162caec not found: ID does not exist"
Jul 09 18:20:47.651907 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.651868    2576 scope.go:117] "RemoveContainer" containerID="08673b9a514124de5782e0c4da4e7e63373ccd4a0b55c9e7d0e4cb3fbc8aca8f"
Jul 09 18:20:47.657987 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.657957    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-68796975c7-ddffs"]
Jul 09 18:20:47.659977 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.659961    2576 scope.go:117] "RemoveContainer" containerID="7b4bbe9dc4ae9ef7754e04d394d6e900bbdeb80faf219d798a79366c52382a74"
Jul 09 18:20:47.671760 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.671713    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" path="/var/lib/kubelet/pods/1b7948b0-afb2-4d45-9e65-bd7dc3175ab6/volumes"
Jul 09 18:20:47.672229 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.672179    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"]
Jul 09 18:20:47.674683 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:47.674662    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-749f64cc84-rf4dw"]
Jul 09 18:20:48.626286 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:48.626237    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:20:49.669633 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:49.669596    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" path="/var/lib/kubelet/pods/dec4c7e9-9050-4477-bbd0-843e5fa80cbd/volumes"
Jul 09 18:20:58.635220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:58.635186    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:20:58.643194 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:20:58.643172    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:21:29.251234 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251198    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"]
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251587    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="storage-initializer"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251598    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="storage-initializer"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251614    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251620    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251632    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251637    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251646    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="storage-initializer"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251651    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="storage-initializer"
Jul 09 18:21:29.251656 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251661    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="llm-d-routing-sidecar"
Jul 09 18:21:29.251956 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251667    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="llm-d-routing-sidecar"
Jul 09 18:21:29.251956 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251723    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="main"
Jul 09 18:21:29.251956 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251733    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7948b0-afb2-4d45-9e65-bd7dc3175ab6" containerName="llm-d-routing-sidecar"
Jul 09 18:21:29.251956 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.251740    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec4c7e9-9050-4477-bbd0-843e5fa80cbd" containerName="main"
Jul 09 18:21:29.254453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.254437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.257217 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.257187    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-zrfd7\""
Jul 09 18:21:29.257439 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.257423    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 09 18:21:29.270513 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.270490    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"]
Jul 09 18:21:29.274064 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.274022    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"]
Jul 09 18:21:29.276668 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.276650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.290709 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.290688    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"]
Jul 09 18:21:29.338594 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cf6a93-22a3-4227-bc61-7819b6fabece-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.338755 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhlx\" (UniqueName: \"kubernetes.io/projected/96cf6a93-22a3-4227-bc61-7819b6fabece-kube-api-access-hqhlx\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.338755 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-dshm\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.338755 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-model-cache\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.338755 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-home\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.338755 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.338969 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tls-certs\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.338969 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.338969 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tmp-dir\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.338969 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.339134 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.338994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-home\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.339134 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.339060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d9z7\" (UniqueName: \"kubernetes.io/projected/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kube-api-access-9d9z7\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.339134 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.339088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.339134 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.339105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440121 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tmp-dir\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.440290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-home\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d9z7\" (UniqueName: \"kubernetes.io/projected/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kube-api-access-9d9z7\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.440290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cf6a93-22a3-4227-bc61-7819b6fabece-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhlx\" (UniqueName: \"kubernetes.io/projected/96cf6a93-22a3-4227-bc61-7819b6fabece-kube-api-access-hqhlx\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-dshm\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-model-cache\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-home\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tls-certs\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440561    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.440653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.441101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-home\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.441101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440562    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tmp-dir\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.441101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-model-cache\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.441101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-home\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.441101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.440925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.441294 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.441105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.442725 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.442703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-dshm\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.442935 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.442912    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.443124 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.443102    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tls-certs\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.443221 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.443203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cf6a93-22a3-4227-bc61-7819b6fabece-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.448825 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.448796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d9z7\" (UniqueName: \"kubernetes.io/projected/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kube-api-access-9d9z7\") pod \"router-with-refs-pd-test-kserve-84dc4d978f-8wd4k\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.449230 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.449212    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhlx\" (UniqueName: \"kubernetes.io/projected/96cf6a93-22a3-4227-bc61-7819b6fabece-kube-api-access-hqhlx\") pod \"router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.564283 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.564254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:29.587835 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.587813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:29.729819 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.729794    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"]
Jul 09 18:21:29.732402 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:21:29.732376    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f9fbc4_24ad_40ce_bae2_8bbd4e537125.slice/crio-119b80c6e77977d105e81ac458ef3a26b5c904f1fef5e1270ba7a35ea4f254a0 WatchSource:0}: Error finding container 119b80c6e77977d105e81ac458ef3a26b5c904f1fef5e1270ba7a35ea4f254a0: Status 404 returned error can't find the container with id 119b80c6e77977d105e81ac458ef3a26b5c904f1fef5e1270ba7a35ea4f254a0
Jul 09 18:21:29.747388 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.747364    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"]
Jul 09 18:21:29.749076 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:21:29.749032    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cf6a93_22a3_4227_bc61_7819b6fabece.slice/crio-9044ce9f9d4ce63cde862a59730a8fca61a7e00dabac8440e24ba39b5830bf61 WatchSource:0}: Error finding container 9044ce9f9d4ce63cde862a59730a8fca61a7e00dabac8440e24ba39b5830bf61: Status 404 returned error can't find the container with id 9044ce9f9d4ce63cde862a59730a8fca61a7e00dabac8440e24ba39b5830bf61
Jul 09 18:21:29.839320 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.839221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" event={"ID":"96cf6a93-22a3-4227-bc61-7819b6fabece","Type":"ContainerStarted","Data":"92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929"}
Jul 09 18:21:29.839320 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.839289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" event={"ID":"96cf6a93-22a3-4227-bc61-7819b6fabece","Type":"ContainerStarted","Data":"9044ce9f9d4ce63cde862a59730a8fca61a7e00dabac8440e24ba39b5830bf61"}
Jul 09 18:21:29.841145 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.841122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerStarted","Data":"06a4b9a8335d6cc506d52cdcc1c4b47a7daba6411fccd6c6ada160be8eeca23d"}
Jul 09 18:21:29.841261 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.841160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerStarted","Data":"119b80c6e77977d105e81ac458ef3a26b5c904f1fef5e1270ba7a35ea4f254a0"}
Jul 09 18:21:29.841261 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:29.841181    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:30.851237 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:30.851103    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerStarted","Data":"0cdd522d9154f10e37d3ca8443e436c62285fde02e57041dc76c278dfcf8f486"}
Jul 09 18:21:33.871957 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:33.871923    2576 generic.go:356] "Generic (PLEG): container finished" podID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerID="92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929" exitCode=0
Jul 09 18:21:33.872382 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:33.871964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" event={"ID":"96cf6a93-22a3-4227-bc61-7819b6fabece","Type":"ContainerDied","Data":"92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929"}
Jul 09 18:21:34.879383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:34.879344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" event={"ID":"96cf6a93-22a3-4227-bc61-7819b6fabece","Type":"ContainerStarted","Data":"cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08"}
Jul 09 18:21:34.881072 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:34.881030    2576 generic.go:356] "Generic (PLEG): container finished" podID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerID="0cdd522d9154f10e37d3ca8443e436c62285fde02e57041dc76c278dfcf8f486" exitCode=0
Jul 09 18:21:34.881196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:34.881076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerDied","Data":"0cdd522d9154f10e37d3ca8443e436c62285fde02e57041dc76c278dfcf8f486"}
Jul 09 18:21:34.909148 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:34.909094    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podStartSLOduration=1.994531448 podStartE2EDuration="5.909077606s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.914546158s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:34.906006252 +0000 UTC m=+2524.436783323" watchObservedRunningTime="2026-07-09 18:21:34.909077606 +0000 UTC m=+2524.439854635"
Jul 09 18:21:35.890657 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:35.890610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerStarted","Data":"7d6d503967c84d504ad498d1545032ae3edbcef6677f1e9018710fd85c74b43f"}
Jul 09 18:21:35.921695 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:35.921633    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podStartSLOduration=2.84628216 podStartE2EDuration="6.921612874s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.075330714s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:35.915950227 +0000 UTC m=+2525.446727283" watchObservedRunningTime="2026-07-09 18:21:35.921612874 +0000 UTC m=+2525.452389917"
Jul 09 18:21:39.565249 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:39.565179    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:39.566784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:39.565258    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:39.566784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:39.566560    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:21:39.583409 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:39.583383    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:21:39.588201 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:39.588170    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:39.588338 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:39.588234    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:21:39.589460 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:39.589433    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:21:43.155147 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:43.155103    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"]
Jul 09 18:21:49.565583 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:49.565522    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:21:49.588555 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:49.588516    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:21:58.180670 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:58.180627    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" containerID="cri-o://9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e" gracePeriod=45
Jul 09 18:21:58.636747 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:58.636702    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 09 18:21:58.957743 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:58.957713    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:21:59.028325 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.028281    2576 generic.go:356] "Generic (PLEG): container finished" podID="629d5b8e-1118-4d76-9ade-835a12406a64" containerID="9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e" exitCode=0
Jul 09 18:21:59.028490 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.028342    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"
Jul 09 18:21:59.028490 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.028367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" event={"ID":"629d5b8e-1118-4d76-9ade-835a12406a64","Type":"ContainerDied","Data":"9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e"}
Jul 09 18:21:59.028490 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.028437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h" event={"ID":"629d5b8e-1118-4d76-9ade-835a12406a64","Type":"ContainerDied","Data":"2a4edb8f41d10d584033d036dd67c00ea2e8d6763a9b6df0bf08883c2a2afc67"}
Jul 09 18:21:59.028490 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.028474    2576 scope.go:117] "RemoveContainer" containerID="9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e"
Jul 09 18:21:59.037748 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.037721    2576 scope.go:117] "RemoveContainer" containerID="48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf"
Jul 09 18:21:59.046183 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.046164    2576 scope.go:117] "RemoveContainer" containerID="9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e"
Jul 09 18:21:59.046483 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:21:59.046466    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e\": container with ID starting with 9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e not found: ID does not exist" containerID="9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e"
Jul 09 18:21:59.046532 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.046491    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e"} err="failed to get container status \"9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e\": rpc error: code = NotFound desc = could not find container \"9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e\": container with ID starting with 9d62c66d138de21bbd4f8628362c5636c42bf907cd2832b8ea981502e326ed7e not found: ID does not exist"
Jul 09 18:21:59.046532 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.046507    2576 scope.go:117] "RemoveContainer" containerID="48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf"
Jul 09 18:21:59.046779 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:21:59.046756    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf\": container with ID starting with 48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf not found: ID does not exist" containerID="48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf"
Jul 09 18:21:59.046892 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.046790    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf"} err="failed to get container status \"48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf\": rpc error: code = NotFound desc = could not find container \"48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf\": container with ID starting with 48fca64755ebc58136d1cdbd1e501609d08c5655b0e48ccdbb0dfa90744b4bcf not found: ID does not exist"
Jul 09 18:21:59.056181 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056154    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-kserve-provision-location\") pod \"629d5b8e-1118-4d76-9ade-835a12406a64\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") "
Jul 09 18:21:59.056305 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056252    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nlkz\" (UniqueName: \"kubernetes.io/projected/629d5b8e-1118-4d76-9ade-835a12406a64-kube-api-access-8nlkz\") pod \"629d5b8e-1118-4d76-9ade-835a12406a64\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") "
Jul 09 18:21:59.056305 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056282    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-model-cache\") pod \"629d5b8e-1118-4d76-9ade-835a12406a64\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") "
Jul 09 18:21:59.056423 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-tmp-dir\") pod \"629d5b8e-1118-4d76-9ade-835a12406a64\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") "
Jul 09 18:21:59.056484 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056454    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-dshm\") pod \"629d5b8e-1118-4d76-9ade-835a12406a64\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") "
Jul 09 18:21:59.056596 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056486    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-home\") pod \"629d5b8e-1118-4d76-9ade-835a12406a64\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") "
Jul 09 18:21:59.056596 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056524    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs\") pod \"629d5b8e-1118-4d76-9ade-835a12406a64\" (UID: \"629d5b8e-1118-4d76-9ade-835a12406a64\") "
Jul 09 18:21:59.056596 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056575    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-model-cache" (OuterVolumeSpecName: "model-cache") pod "629d5b8e-1118-4d76-9ade-835a12406a64" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:59.056848 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.056830    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:21:59.057277 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.057224    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-home" (OuterVolumeSpecName: "home") pod "629d5b8e-1118-4d76-9ade-835a12406a64" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:59.059066 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.058990    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-dshm" (OuterVolumeSpecName: "dshm") pod "629d5b8e-1118-4d76-9ade-835a12406a64" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:59.059519 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.059483    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629d5b8e-1118-4d76-9ade-835a12406a64-kube-api-access-8nlkz" (OuterVolumeSpecName: "kube-api-access-8nlkz") pod "629d5b8e-1118-4d76-9ade-835a12406a64" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64"). InnerVolumeSpecName "kube-api-access-8nlkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:59.059616 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.059583    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "629d5b8e-1118-4d76-9ade-835a12406a64" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:59.068101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.068077    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "629d5b8e-1118-4d76-9ade-835a12406a64" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:59.127084 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.127021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "629d5b8e-1118-4d76-9ade-835a12406a64" (UID: "629d5b8e-1118-4d76-9ade-835a12406a64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:59.158413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.158372    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nlkz\" (UniqueName: \"kubernetes.io/projected/629d5b8e-1118-4d76-9ade-835a12406a64-kube-api-access-8nlkz\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:21:59.158413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.158402    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:21:59.158413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.158412    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:21:59.158413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.158421    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:21:59.158716 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.158428    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d5b8e-1118-4d76-9ade-835a12406a64-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:21:59.158716 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.158437    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d5b8e-1118-4d76-9ade-835a12406a64-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:21:59.351818 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.351786    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"]
Jul 09 18:21:59.354912 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.354889    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-m5r6h"]
Jul 09 18:21:59.565316 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.565271    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:21:59.588381 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.588288    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:21:59.670051 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:21:59.670006    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" path="/var/lib/kubelet/pods/629d5b8e-1118-4d76-9ade-835a12406a64/volumes"
Jul 09 18:22:09.565098 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:09.565028    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:22:09.588363 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:09.588323    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:22:19.565735 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:19.565681    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:22:19.589087 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:19.589031    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:22:29.564871 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:29.564821    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:22:29.588597 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:29.588561    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:22:37.118349 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.118311    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"]
Jul 09 18:22:37.118923 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.118902    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main"
Jul 09 18:22:37.118977 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.118929    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main"
Jul 09 18:22:37.118977 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.118945    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="storage-initializer"
Jul 09 18:22:37.118977 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.118954    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="storage-initializer"
Jul 09 18:22:37.119114 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.119101    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="629d5b8e-1118-4d76-9ade-835a12406a64" containerName="main"
Jul 09 18:22:37.122292 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.122262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.125489 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.125465    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 18:22:37.137863 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.137838    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"]
Jul 09 18:22:37.245976 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.245935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l445l\" (UniqueName: \"kubernetes.io/projected/60b4e716-77fc-4bf1-a249-fad5d030e314-kube-api-access-l445l\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.246162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.245988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-home\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.246162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.246087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-kserve-provision-location\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.246162 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.246135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-tmp-dir\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.246307 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.246189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-dshm\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.246307 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.246263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4e716-77fc-4bf1-a249-fad5d030e314-tls-certs\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.246382 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.246321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-model-cache\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.346953 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.346920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l445l\" (UniqueName: \"kubernetes.io/projected/60b4e716-77fc-4bf1-a249-fad5d030e314-kube-api-access-l445l\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347153 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.346965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-home\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347153 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-kserve-provision-location\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347283 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-tmp-dir\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347283 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-dshm\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4e716-77fc-4bf1-a249-fad5d030e314-tls-certs\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-model-cache\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347510 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-home\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347666 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-tmp-dir\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.347751 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.347721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-model-cache\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.348289 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.348268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-kserve-provision-location\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.350407 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.350380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-dshm\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.350931 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.350907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4e716-77fc-4bf1-a249-fad5d030e314-tls-certs\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.363845 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.363821    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l445l\" (UniqueName: \"kubernetes.io/projected/60b4e716-77fc-4bf1-a249-fad5d030e314-kube-api-access-l445l\") pod \"stop-feature-test-kserve-745fc78d48-mrzcn\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.433850 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.433768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:37.583406 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:37.583375    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"]
Jul 09 18:22:37.586577 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:22:37.586543    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b4e716_77fc_4bf1_a249_fad5d030e314.slice/crio-f74970bf7d97e5ba5820d57c662e62338b5fc8f9b9d981a4e96e1a0fb2617970 WatchSource:0}: Error finding container f74970bf7d97e5ba5820d57c662e62338b5fc8f9b9d981a4e96e1a0fb2617970: Status 404 returned error can't find the container with id f74970bf7d97e5ba5820d57c662e62338b5fc8f9b9d981a4e96e1a0fb2617970
Jul 09 18:22:38.254025 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:38.253983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" event={"ID":"60b4e716-77fc-4bf1-a249-fad5d030e314","Type":"ContainerStarted","Data":"6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385"}
Jul 09 18:22:38.254505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:38.254074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" event={"ID":"60b4e716-77fc-4bf1-a249-fad5d030e314","Type":"ContainerStarted","Data":"f74970bf7d97e5ba5820d57c662e62338b5fc8f9b9d981a4e96e1a0fb2617970"}
Jul 09 18:22:39.564972 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:39.564838    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:22:39.588825 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:39.588791    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:22:41.274027 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:41.273986    2576 generic.go:356] "Generic (PLEG): container finished" podID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerID="6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385" exitCode=0
Jul 09 18:22:41.274496 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:41.274070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" event={"ID":"60b4e716-77fc-4bf1-a249-fad5d030e314","Type":"ContainerDied","Data":"6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385"}
Jul 09 18:22:42.283110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:42.283072    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" event={"ID":"60b4e716-77fc-4bf1-a249-fad5d030e314","Type":"ContainerStarted","Data":"add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f"}
Jul 09 18:22:42.308160 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:42.308103    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podStartSLOduration=1.826891517 podStartE2EDuration="5.308082176s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.481190659s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:22:42.305349441 +0000 UTC m=+2591.836126497" watchObservedRunningTime="2026-07-09 18:22:42.308082176 +0000 UTC m=+2591.838859207"
Jul 09 18:22:42.433896 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:42.433856    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:47.434231 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:47.434184    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:22:47.435984 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:47.435945    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:22:49.565534 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:49.565448    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:22:49.588312 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:49.588276    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:22:57.435110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:57.435001    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:22:59.565195 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:59.565149    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:22:59.588537 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:22:59.588503    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:23:07.434613 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:07.434564    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:23:09.565452 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:09.565397    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:23:09.588288 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:09.588249    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:23:17.434787 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:17.434747    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:23:19.565173 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:19.565128    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:23:19.588405 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:19.588362    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:23:27.434232 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:27.434190    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:23:29.564803 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:29.564765    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:23:29.588859 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:29.588820    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:23:37.434431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:37.434389    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:23:39.564961 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:39.564914    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:23:39.588519 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:39.588220    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:23:47.435348 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:47.435295    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:23:49.565505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:49.565459    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:23:49.588621 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:49.588583    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:23:57.434723 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:57.434676    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:23:59.564847 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:59.564806    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:23:59.588459 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:23:59.588427    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:24:07.435069 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:07.435012    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:24:09.564838 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:09.564784    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Jul 09 18:24:09.588986 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:09.588943    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 09 18:24:17.434653 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:17.434608    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:24:19.579987 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:19.579958    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:24:19.591937 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:19.591912    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:24:19.599634 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:19.599611    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:24:19.608390 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:19.608357    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:24:27.434593 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:27.434479    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:24:31.180123 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:31.180089    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:24:31.184889 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:31.184869    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:24:34.574373 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:34.574337    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"]
Jul 09 18:24:34.588431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:34.588397    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"]
Jul 09 18:24:37.434942 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:37.434889    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:24:47.435131 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:47.435084    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:24:49.603874 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:49.603826    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main" containerID="cri-o://cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08" gracePeriod=45
Jul 09 18:24:49.613803 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:49.613758    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main" containerID="cri-o://7d6d503967c84d504ad498d1545032ae3edbcef6677f1e9018710fd85c74b43f" gracePeriod=45
Jul 09 18:24:50.723871 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:50.723697    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="llm-d-routing-sidecar" containerID="cri-o://06a4b9a8335d6cc506d52cdcc1c4b47a7daba6411fccd6c6ada160be8eeca23d" gracePeriod=44
Jul 09 18:24:50.898813 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:50.898790    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:24:51.001752 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.001721    2576 generic.go:356] "Generic (PLEG): container finished" podID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerID="cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08" exitCode=0
Jul 09 18:24:51.001910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.001822    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"
Jul 09 18:24:51.001910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.001846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" event={"ID":"96cf6a93-22a3-4227-bc61-7819b6fabece","Type":"ContainerDied","Data":"cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08"}
Jul 09 18:24:51.002005 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.001940    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4" event={"ID":"96cf6a93-22a3-4227-bc61-7819b6fabece","Type":"ContainerDied","Data":"9044ce9f9d4ce63cde862a59730a8fca61a7e00dabac8440e24ba39b5830bf61"}
Jul 09 18:24:51.002005 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.001972    2576 scope.go:117] "RemoveContainer" containerID="cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08"
Jul 09 18:24:51.003835 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.003817    2576 generic.go:356] "Generic (PLEG): container finished" podID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerID="7d6d503967c84d504ad498d1545032ae3edbcef6677f1e9018710fd85c74b43f" exitCode=0
Jul 09 18:24:51.003835 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.003832    2576 generic.go:356] "Generic (PLEG): container finished" podID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerID="06a4b9a8335d6cc506d52cdcc1c4b47a7daba6411fccd6c6ada160be8eeca23d" exitCode=0
Jul 09 18:24:51.003989 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.003884    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerDied","Data":"7d6d503967c84d504ad498d1545032ae3edbcef6677f1e9018710fd85c74b43f"}
Jul 09 18:24:51.003989 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.003922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerDied","Data":"06a4b9a8335d6cc506d52cdcc1c4b47a7daba6411fccd6c6ada160be8eeca23d"}
Jul 09 18:24:51.004205 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.004188    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:24:51.010786 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.010772    2576 scope.go:117] "RemoveContainer" containerID="92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929"
Jul 09 18:24:51.018502 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.018487    2576 scope.go:117] "RemoveContainer" containerID="cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08"
Jul 09 18:24:51.018744 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:24:51.018724    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08\": container with ID starting with cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08 not found: ID does not exist" containerID="cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08"
Jul 09 18:24:51.018789 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.018754    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08"} err="failed to get container status \"cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08\": rpc error: code = NotFound desc = could not find container \"cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08\": container with ID starting with cc556eab35e3763f235c1d1c71c30445e88a86195add5cc490573f6220023a08 not found: ID does not exist"
Jul 09 18:24:51.018789 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.018770    2576 scope.go:117] "RemoveContainer" containerID="92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929"
Jul 09 18:24:51.018970 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:24:51.018955    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929\": container with ID starting with 92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929 not found: ID does not exist" containerID="92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929"
Jul 09 18:24:51.019011 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.018977    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929"} err="failed to get container status \"92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929\": rpc error: code = NotFound desc = could not find container \"92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929\": container with ID starting with 92cdbc1261f7bf771befa1197d687f138a7ad6a52a25314ae20fbff993a9d929 not found: ID does not exist"
Jul 09 18:24:51.025330 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.025308    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-dshm\") pod \"96cf6a93-22a3-4227-bc61-7819b6fabece\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") "
Jul 09 18:24:51.025540 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.025523    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-model-cache\") pod \"96cf6a93-22a3-4227-bc61-7819b6fabece\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") "
Jul 09 18:24:51.025677 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.025663    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-tmp-dir\") pod \"96cf6a93-22a3-4227-bc61-7819b6fabece\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") "
Jul 09 18:24:51.025818 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.025794    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-model-cache" (OuterVolumeSpecName: "model-cache") pod "96cf6a93-22a3-4227-bc61-7819b6fabece" (UID: "96cf6a93-22a3-4227-bc61-7819b6fabece"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.025999 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.025984    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cf6a93-22a3-4227-bc61-7819b6fabece-tls-certs\") pod \"96cf6a93-22a3-4227-bc61-7819b6fabece\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") "
Jul 09 18:24:51.026486 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.026468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-kserve-provision-location\") pod \"96cf6a93-22a3-4227-bc61-7819b6fabece\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") "
Jul 09 18:24:51.026671 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.026656    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqhlx\" (UniqueName: \"kubernetes.io/projected/96cf6a93-22a3-4227-bc61-7819b6fabece-kube-api-access-hqhlx\") pod \"96cf6a93-22a3-4227-bc61-7819b6fabece\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") "
Jul 09 18:24:51.027207 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.027190    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-home\") pod \"96cf6a93-22a3-4227-bc61-7819b6fabece\" (UID: \"96cf6a93-22a3-4227-bc61-7819b6fabece\") "
Jul 09 18:24:51.027670 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.027655    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.028435 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.028412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-home" (OuterVolumeSpecName: "home") pod "96cf6a93-22a3-4227-bc61-7819b6fabece" (UID: "96cf6a93-22a3-4227-bc61-7819b6fabece"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.038140 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.038102    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cf6a93-22a3-4227-bc61-7819b6fabece-kube-api-access-hqhlx" (OuterVolumeSpecName: "kube-api-access-hqhlx") pod "96cf6a93-22a3-4227-bc61-7819b6fabece" (UID: "96cf6a93-22a3-4227-bc61-7819b6fabece"). InnerVolumeSpecName "kube-api-access-hqhlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:51.039159 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.039127    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cf6a93-22a3-4227-bc61-7819b6fabece-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96cf6a93-22a3-4227-bc61-7819b6fabece" (UID: "96cf6a93-22a3-4227-bc61-7819b6fabece"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:51.041668 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.041626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-dshm" (OuterVolumeSpecName: "dshm") pod "96cf6a93-22a3-4227-bc61-7819b6fabece" (UID: "96cf6a93-22a3-4227-bc61-7819b6fabece"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.059649 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.059240    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96cf6a93-22a3-4227-bc61-7819b6fabece" (UID: "96cf6a93-22a3-4227-bc61-7819b6fabece"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.116335 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.116297    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96cf6a93-22a3-4227-bc61-7819b6fabece" (UID: "96cf6a93-22a3-4227-bc61-7819b6fabece"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.128896 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.128869    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tls-certs\") pod \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") "
Jul 09 18:24:51.129069 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.128948    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-home\") pod \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") "
Jul 09 18:24:51.129069 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.128989    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d9z7\" (UniqueName: \"kubernetes.io/projected/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kube-api-access-9d9z7\") pod \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") "
Jul 09 18:24:51.129206 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-model-cache\") pod \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") "
Jul 09 18:24:51.129206 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129119    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tmp-dir\") pod \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") "
Jul 09 18:24:51.129206 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129162    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-dshm\") pod \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") "
Jul 09 18:24:51.129206 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kserve-provision-location\") pod \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\" (UID: \"08f9fbc4-24ad-40ce-bae2-8bbd4e537125\") "
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129472    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-model-cache" (OuterVolumeSpecName: "model-cache") pod "08f9fbc4-24ad-40ce-bae2-8bbd4e537125" (UID: "08f9fbc4-24ad-40ce-bae2-8bbd4e537125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129530    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129547    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129561    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129574    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cf6a93-22a3-4227-bc61-7819b6fabece-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129589    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cf6a93-22a3-4227-bc61-7819b6fabece-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.129604    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhlx\" (UniqueName: \"kubernetes.io/projected/96cf6a93-22a3-4227-bc61-7819b6fabece-kube-api-access-hqhlx\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.132824 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.132770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-home" (OuterVolumeSpecName: "home") pod "08f9fbc4-24ad-40ce-bae2-8bbd4e537125" (UID: "08f9fbc4-24ad-40ce-bae2-8bbd4e537125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.133717 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.133690    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08f9fbc4-24ad-40ce-bae2-8bbd4e537125" (UID: "08f9fbc4-24ad-40ce-bae2-8bbd4e537125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:51.134237 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.134086    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kube-api-access-9d9z7" (OuterVolumeSpecName: "kube-api-access-9d9z7") pod "08f9fbc4-24ad-40ce-bae2-8bbd4e537125" (UID: "08f9fbc4-24ad-40ce-bae2-8bbd4e537125"). InnerVolumeSpecName "kube-api-access-9d9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:51.139293 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.139261    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-dshm" (OuterVolumeSpecName: "dshm") pod "08f9fbc4-24ad-40ce-bae2-8bbd4e537125" (UID: "08f9fbc4-24ad-40ce-bae2-8bbd4e537125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.154458 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.154405    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08f9fbc4-24ad-40ce-bae2-8bbd4e537125" (UID: "08f9fbc4-24ad-40ce-bae2-8bbd4e537125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.217955 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.217904    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08f9fbc4-24ad-40ce-bae2-8bbd4e537125" (UID: "08f9fbc4-24ad-40ce-bae2-8bbd4e537125"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:51.230775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.230626    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.230775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.230655    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.230775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.230672    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.230775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.230687    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.230775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.230703    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.230775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.230716    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d9z7\" (UniqueName: \"kubernetes.io/projected/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-kube-api-access-9d9z7\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.230775 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.230732    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f9fbc4-24ad-40ce-bae2-8bbd4e537125-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:24:51.327457 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.327428    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"]
Jul 09 18:24:51.333793 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.333767    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-84464f6744-knbm4"]
Jul 09 18:24:51.672152 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:51.672127    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" path="/var/lib/kubelet/pods/96cf6a93-22a3-4227-bc61-7819b6fabece/volumes"
Jul 09 18:24:52.011935 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:52.011844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k" event={"ID":"08f9fbc4-24ad-40ce-bae2-8bbd4e537125","Type":"ContainerDied","Data":"119b80c6e77977d105e81ac458ef3a26b5c904f1fef5e1270ba7a35ea4f254a0"}
Jul 09 18:24:52.011935 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:52.011881    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"
Jul 09 18:24:52.011935 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:52.011920    2576 scope.go:117] "RemoveContainer" containerID="7d6d503967c84d504ad498d1545032ae3edbcef6677f1e9018710fd85c74b43f"
Jul 09 18:24:52.021281 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:52.021262    2576 scope.go:117] "RemoveContainer" containerID="0cdd522d9154f10e37d3ca8443e436c62285fde02e57041dc76c278dfcf8f486"
Jul 09 18:24:52.029491 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:52.029471    2576 scope.go:117] "RemoveContainer" containerID="06a4b9a8335d6cc506d52cdcc1c4b47a7daba6411fccd6c6ada160be8eeca23d"
Jul 09 18:24:52.035969 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:52.035947    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"]
Jul 09 18:24:52.042091 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:52.042071    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-84dc4d978f-8wd4k"]
Jul 09 18:24:53.669784 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:53.669756    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" path="/var/lib/kubelet/pods/08f9fbc4-24ad-40ce-bae2-8bbd4e537125/volumes"
Jul 09 18:24:57.444620 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:57.444590    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:24:57.452686 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:24:57.452663    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:25:07.729470 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.729436    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"]
Jul 09 18:25:07.730007 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.729989    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="storage-initializer"
Jul 09 18:25:07.730082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730010    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="storage-initializer"
Jul 09 18:25:07.730082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730066    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="storage-initializer"
Jul 09 18:25:07.730082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730077    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="storage-initializer"
Jul 09 18:25:07.730182 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730097    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="llm-d-routing-sidecar"
Jul 09 18:25:07.730182 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730106    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="llm-d-routing-sidecar"
Jul 09 18:25:07.730182 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730126    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main"
Jul 09 18:25:07.730182 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730135    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main"
Jul 09 18:25:07.730182 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730148    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main"
Jul 09 18:25:07.730182 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730156    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main"
Jul 09 18:25:07.730369 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730243    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="main"
Jul 09 18:25:07.730369 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730258    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f9fbc4-24ad-40ce-bae2-8bbd4e537125" containerName="llm-d-routing-sidecar"
Jul 09 18:25:07.730369 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.730267    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cf6a93-22a3-4227-bc61-7819b6fabece" containerName="main"
Jul 09 18:25:07.734337 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.734322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.737319 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.737295    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 09 18:25:07.748379 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.748359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"]
Jul 09 18:25:07.766539 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.766513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.766635 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.766567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.766682 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.766629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.766682 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.766673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.766758 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.766723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.766758 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.766742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.766827 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.766808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7n4k\" (UniqueName: \"kubernetes.io/projected/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kube-api-access-n7n4k\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867333 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7n4k\" (UniqueName: \"kubernetes.io/projected/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kube-api-access-n7n4k\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867457 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867457 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867457 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867457 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867663 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867663 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867867 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867844    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867935 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.867976 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867956    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.868014 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.867963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.869705 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.869687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.869953 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.869937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:07.883023 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:07.883005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7n4k\" (UniqueName: \"kubernetes.io/projected/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kube-api-access-n7n4k\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:08.044623 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:08.044563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:08.191285 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:08.191253    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"]
Jul 09 18:25:08.195955 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:25:08.195927    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c253c85_6d6d_4152_99a6_e0b600b2e52c.slice/crio-cf2574b5a48ccf915911d62f39546e89f25e87355c9e8aa1a8e52ebeb4f3a73e WatchSource:0}: Error finding container cf2574b5a48ccf915911d62f39546e89f25e87355c9e8aa1a8e52ebeb4f3a73e: Status 404 returned error can't find the container with id cf2574b5a48ccf915911d62f39546e89f25e87355c9e8aa1a8e52ebeb4f3a73e
Jul 09 18:25:09.098789 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:09.098707    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" event={"ID":"1c253c85-6d6d-4152-99a6-e0b600b2e52c","Type":"ContainerStarted","Data":"8377474bfbceb266b4ce095bc7857b1472cc440689cd3670dfd593694b86d81e"}
Jul 09 18:25:09.099419 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:09.098820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" event={"ID":"1c253c85-6d6d-4152-99a6-e0b600b2e52c","Type":"ContainerStarted","Data":"cf2574b5a48ccf915911d62f39546e89f25e87355c9e8aa1a8e52ebeb4f3a73e"}
Jul 09 18:25:12.115068 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:12.114982    2576 generic.go:356] "Generic (PLEG): container finished" podID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerID="8377474bfbceb266b4ce095bc7857b1472cc440689cd3670dfd593694b86d81e" exitCode=0
Jul 09 18:25:12.115385 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:12.115074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" event={"ID":"1c253c85-6d6d-4152-99a6-e0b600b2e52c","Type":"ContainerDied","Data":"8377474bfbceb266b4ce095bc7857b1472cc440689cd3670dfd593694b86d81e"}
Jul 09 18:25:13.121609 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:13.121575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" event={"ID":"1c253c85-6d6d-4152-99a6-e0b600b2e52c","Type":"ContainerStarted","Data":"d0c50c6f4e29047f245b50edd5576ad172d3e3db8554a65388a0702e9510dd30"}
Jul 09 18:25:13.153862 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:13.153811    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podStartSLOduration=2.584746477 podStartE2EDuration="6.153796475s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.569049998s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:13.152473704 +0000 UTC m=+2742.683250735" watchObservedRunningTime="2026-07-09 18:25:13.153796475 +0000 UTC m=+2742.684573507"
Jul 09 18:25:14.045346 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:14.045309    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:18.045007 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:18.044969    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:25:18.046721 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:18.046690    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:25:27.384712 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:27.384677    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"]
Jul 09 18:25:28.045383 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:28.045342    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:25:38.045982 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:38.045932    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:25:42.411247 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:42.411211    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" containerID="cri-o://add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f" gracePeriod=45
Jul 09 18:25:42.445259 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:42.445211    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 18:25:43.183981 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.183927    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:25:43.277395 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.277362    2576 generic.go:356] "Generic (PLEG): container finished" podID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerID="add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f" exitCode=0
Jul 09 18:25:43.277562 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.277420    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"
Jul 09 18:25:43.277562 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.277427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" event={"ID":"60b4e716-77fc-4bf1-a249-fad5d030e314","Type":"ContainerDied","Data":"add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f"}
Jul 09 18:25:43.277562 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.277490    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn" event={"ID":"60b4e716-77fc-4bf1-a249-fad5d030e314","Type":"ContainerDied","Data":"f74970bf7d97e5ba5820d57c662e62338b5fc8f9b9d981a4e96e1a0fb2617970"}
Jul 09 18:25:43.277562 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.277523    2576 scope.go:117] "RemoveContainer" containerID="add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f"
Jul 09 18:25:43.286051 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.286024    2576 scope.go:117] "RemoveContainer" containerID="6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385"
Jul 09 18:25:43.294166 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.294148    2576 scope.go:117] "RemoveContainer" containerID="add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f"
Jul 09 18:25:43.294450 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:25:43.294432    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f\": container with ID starting with add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f not found: ID does not exist" containerID="add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f"
Jul 09 18:25:43.294509 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.294461    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f"} err="failed to get container status \"add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f\": rpc error: code = NotFound desc = could not find container \"add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f\": container with ID starting with add79ab647dfbd9818551ca7ba81adeb4ddd466e0fca69e87937b3a610f43a2f not found: ID does not exist"
Jul 09 18:25:43.294509 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.294479    2576 scope.go:117] "RemoveContainer" containerID="6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385"
Jul 09 18:25:43.294743 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:25:43.294728    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385\": container with ID starting with 6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385 not found: ID does not exist" containerID="6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385"
Jul 09 18:25:43.294795 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.294747    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385"} err="failed to get container status \"6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385\": rpc error: code = NotFound desc = could not find container \"6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385\": container with ID starting with 6085c7f6fe15d584fc3c6af09c133bde54cb18fb416784ed5143568bb33b8385 not found: ID does not exist"
Jul 09 18:25:43.298187 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298171    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-dshm\") pod \"60b4e716-77fc-4bf1-a249-fad5d030e314\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") "
Jul 09 18:25:43.298247 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298201    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4e716-77fc-4bf1-a249-fad5d030e314-tls-certs\") pod \"60b4e716-77fc-4bf1-a249-fad5d030e314\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") "
Jul 09 18:25:43.298247 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298238    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-home\") pod \"60b4e716-77fc-4bf1-a249-fad5d030e314\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") "
Jul 09 18:25:43.298365 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298261    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-model-cache\") pod \"60b4e716-77fc-4bf1-a249-fad5d030e314\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") "
Jul 09 18:25:43.298365 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298314    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-kserve-provision-location\") pod \"60b4e716-77fc-4bf1-a249-fad5d030e314\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") "
Jul 09 18:25:43.298473 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298371    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-tmp-dir\") pod \"60b4e716-77fc-4bf1-a249-fad5d030e314\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") "
Jul 09 18:25:43.298473 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298392    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l445l\" (UniqueName: \"kubernetes.io/projected/60b4e716-77fc-4bf1-a249-fad5d030e314-kube-api-access-l445l\") pod \"60b4e716-77fc-4bf1-a249-fad5d030e314\" (UID: \"60b4e716-77fc-4bf1-a249-fad5d030e314\") "
Jul 09 18:25:43.298606 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298564    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-model-cache" (OuterVolumeSpecName: "model-cache") pod "60b4e716-77fc-4bf1-a249-fad5d030e314" (UID: "60b4e716-77fc-4bf1-a249-fad5d030e314"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:43.298777 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298756    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:25:43.298902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.298879    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-home" (OuterVolumeSpecName: "home") pod "60b4e716-77fc-4bf1-a249-fad5d030e314" (UID: "60b4e716-77fc-4bf1-a249-fad5d030e314"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:43.300350 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.300315    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-dshm" (OuterVolumeSpecName: "dshm") pod "60b4e716-77fc-4bf1-a249-fad5d030e314" (UID: "60b4e716-77fc-4bf1-a249-fad5d030e314"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:43.300459 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.300440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b4e716-77fc-4bf1-a249-fad5d030e314-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60b4e716-77fc-4bf1-a249-fad5d030e314" (UID: "60b4e716-77fc-4bf1-a249-fad5d030e314"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:43.300912 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.300893    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b4e716-77fc-4bf1-a249-fad5d030e314-kube-api-access-l445l" (OuterVolumeSpecName: "kube-api-access-l445l") pod "60b4e716-77fc-4bf1-a249-fad5d030e314" (UID: "60b4e716-77fc-4bf1-a249-fad5d030e314"). InnerVolumeSpecName "kube-api-access-l445l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:43.315402 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.315365    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60b4e716-77fc-4bf1-a249-fad5d030e314" (UID: "60b4e716-77fc-4bf1-a249-fad5d030e314"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:43.361736 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.361708    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60b4e716-77fc-4bf1-a249-fad5d030e314" (UID: "60b4e716-77fc-4bf1-a249-fad5d030e314"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:43.399795 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.399766    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l445l\" (UniqueName: \"kubernetes.io/projected/60b4e716-77fc-4bf1-a249-fad5d030e314-kube-api-access-l445l\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:25:43.399795 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.399791    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:25:43.399934 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.399805    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b4e716-77fc-4bf1-a249-fad5d030e314-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:25:43.399934 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.399819    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:25:43.399934 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.399830    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:25:43.399934 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.399841    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b4e716-77fc-4bf1-a249-fad5d030e314-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:25:43.603538 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.603504    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"]
Jul 09 18:25:43.606637 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.606615    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-745fc78d48-mrzcn"]
Jul 09 18:25:43.670344 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:43.670316    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" path="/var/lib/kubelet/pods/60b4e716-77fc-4bf1-a249-fad5d030e314/volumes"
Jul 09 18:25:48.045110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:48.045067    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:25:58.044962 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:58.044875    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:25:59.885441 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.885409    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"]
Jul 09 18:25:59.885805 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.885793    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="storage-initializer"
Jul 09 18:25:59.885850 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.885807    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="storage-initializer"
Jul 09 18:25:59.885850 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.885826    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main"
Jul 09 18:25:59.885850 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.885831    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main"
Jul 09 18:25:59.885943 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.885897    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b4e716-77fc-4bf1-a249-fad5d030e314" containerName="main"
Jul 09 18:25:59.888075 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.888057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.891220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.891199    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 09 18:25:59.903703 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.903683    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"]
Jul 09 18:25:59.956426 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b214b-35d0-4388-8391-16551267b310-cabundle-cert\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.956577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-tmp-dir\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.956577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.956577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzln\" (UniqueName: \"kubernetes.io/projected/dc6b214b-35d0-4388-8391-16551267b310-kube-api-access-6dzln\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.956686 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-dshm\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.956686 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b214b-35d0-4388-8391-16551267b310-tls-certs\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.956686 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-home\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:25:59.956782 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:25:59.956715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-model-cache\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057314 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b214b-35d0-4388-8391-16551267b310-cabundle-cert\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057487 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-tmp-dir\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057487 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057487 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzln\" (UniqueName: \"kubernetes.io/projected/dc6b214b-35d0-4388-8391-16551267b310-kube-api-access-6dzln\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057487 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-dshm\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057487 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b214b-35d0-4388-8391-16551267b310-tls-certs\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057767 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-home\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057767 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-model-cache\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057767 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057735    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-tmp-dir\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.057951 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-model-cache\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.058005 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.057959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.058097 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.058079    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-home\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.058215 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.058197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b214b-35d0-4388-8391-16551267b310-cabundle-cert\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.059683 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.059658    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-dshm\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.059945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.059925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b214b-35d0-4388-8391-16551267b310-tls-certs\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.072169 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.072144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzln\" (UniqueName: \"kubernetes.io/projected/dc6b214b-35d0-4388-8391-16551267b310-kube-api-access-6dzln\") pod \"lora-single-adapter-test-kserve-84645dc545-rjkj5\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.198984 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.198897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:00.350152 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.350125    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"]
Jul 09 18:26:00.350867 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:26:00.350842    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6b214b_35d0_4388_8391_16551267b310.slice/crio-9ba79fa0344c06c4b4687dfe3d091f6d703e44c6740ac36fa447452cf54cc13d WatchSource:0}: Error finding container 9ba79fa0344c06c4b4687dfe3d091f6d703e44c6740ac36fa447452cf54cc13d: Status 404 returned error can't find the container with id 9ba79fa0344c06c4b4687dfe3d091f6d703e44c6740ac36fa447452cf54cc13d
Jul 09 18:26:00.365364 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:00.365335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" event={"ID":"dc6b214b-35d0-4388-8391-16551267b310","Type":"ContainerStarted","Data":"9ba79fa0344c06c4b4687dfe3d091f6d703e44c6740ac36fa447452cf54cc13d"}
Jul 09 18:26:01.376328 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:01.376286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" event={"ID":"dc6b214b-35d0-4388-8391-16551267b310","Type":"ContainerStarted","Data":"6875a68cc904e787916a55d441aeeaca10eaa35bffa1ab9d6eec3bf9866037f3"}
Jul 09 18:26:05.399471 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:05.399437    2576 generic.go:356] "Generic (PLEG): container finished" podID="dc6b214b-35d0-4388-8391-16551267b310" containerID="6875a68cc904e787916a55d441aeeaca10eaa35bffa1ab9d6eec3bf9866037f3" exitCode=0
Jul 09 18:26:05.399817 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:05.399495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" event={"ID":"dc6b214b-35d0-4388-8391-16551267b310","Type":"ContainerDied","Data":"6875a68cc904e787916a55d441aeeaca10eaa35bffa1ab9d6eec3bf9866037f3"}
Jul 09 18:26:06.408380 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:06.408341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" event={"ID":"dc6b214b-35d0-4388-8391-16551267b310","Type":"ContainerStarted","Data":"54850454fe2f93a5a8e086ed4f9a1e5a62b7717da4bab23f39adc5d765b3691c"}
Jul 09 18:26:06.433714 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:06.433663    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podStartSLOduration=2.681338689 podStartE2EDuration="7.43364756s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.752308871s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:06.433317074 +0000 UTC m=+2795.964094135" watchObservedRunningTime="2026-07-09 18:26:06.43364756 +0000 UTC m=+2795.964424591"
Jul 09 18:26:07.199612 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:07.199569    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:08.045056 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:08.044994    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:26:10.199979 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:10.199948    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:26:10.201130 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:10.201097    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 09 18:26:18.045002 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:18.044950    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:26:20.199316 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:20.199267    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 09 18:26:28.045908 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:28.045864    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:26:30.199554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:30.199520    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 09 18:26:38.044979 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:38.044927    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:26:40.200173 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:40.200133    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 09 18:26:48.045687 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:48.045638    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:26:50.200304 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:50.200257    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 09 18:26:58.045730 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:26:58.045682    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:27:00.199829 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:00.199787    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 09 18:27:08.055085 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:08.055053    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:27:08.062828 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:08.062803    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:27:10.199900 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:10.199832    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 09 18:27:12.764772 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:12.764735    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"]
Jul 09 18:27:20.209443 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:20.209415    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:27:20.217148 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:20.217128    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:27:27.789521 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:27.789430    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" containerID="cri-o://d0c50c6f4e29047f245b50edd5576ad172d3e3db8554a65388a0702e9510dd30" gracePeriod=45
Jul 09 18:27:28.056018 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.055931    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 09 18:27:28.859908 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.859872    2576 generic.go:356] "Generic (PLEG): container finished" podID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerID="d0c50c6f4e29047f245b50edd5576ad172d3e3db8554a65388a0702e9510dd30" exitCode=0
Jul 09 18:27:28.860310 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.859947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" event={"ID":"1c253c85-6d6d-4152-99a6-e0b600b2e52c","Type":"ContainerDied","Data":"d0c50c6f4e29047f245b50edd5576ad172d3e3db8554a65388a0702e9510dd30"}
Jul 09 18:27:28.860310 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.860000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf" event={"ID":"1c253c85-6d6d-4152-99a6-e0b600b2e52c","Type":"ContainerDied","Data":"cf2574b5a48ccf915911d62f39546e89f25e87355c9e8aa1a8e52ebeb4f3a73e"}
Jul 09 18:27:28.860310 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.860017    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf2574b5a48ccf915911d62f39546e89f25e87355c9e8aa1a8e52ebeb4f3a73e"
Jul 09 18:27:28.861659 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.861637    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:27:28.942678 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-dshm\") pod \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") "
Jul 09 18:27:28.942678 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942657    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-model-cache\") pod \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") "
Jul 09 18:27:28.942888 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942697    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7n4k\" (UniqueName: \"kubernetes.io/projected/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kube-api-access-n7n4k\") pod \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") "
Jul 09 18:27:28.942888 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942725    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-home\") pod \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") "
Jul 09 18:27:28.942888 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942752    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tmp-dir\") pod \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") "
Jul 09 18:27:28.942888 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942793    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tls-certs\") pod \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") "
Jul 09 18:27:28.942888 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942856    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kserve-provision-location\") pod \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\" (UID: \"1c253c85-6d6d-4152-99a6-e0b600b2e52c\") "
Jul 09 18:27:28.943189 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.942916    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c253c85-6d6d-4152-99a6-e0b600b2e52c" (UID: "1c253c85-6d6d-4152-99a6-e0b600b2e52c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:28.943253 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.943226    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:27:28.943463 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.943433    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-home" (OuterVolumeSpecName: "home") pod "1c253c85-6d6d-4152-99a6-e0b600b2e52c" (UID: "1c253c85-6d6d-4152-99a6-e0b600b2e52c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:28.944925 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.944901    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-dshm" (OuterVolumeSpecName: "dshm") pod "1c253c85-6d6d-4152-99a6-e0b600b2e52c" (UID: "1c253c85-6d6d-4152-99a6-e0b600b2e52c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:28.945290 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.945266    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kube-api-access-n7n4k" (OuterVolumeSpecName: "kube-api-access-n7n4k") pod "1c253c85-6d6d-4152-99a6-e0b600b2e52c" (UID: "1c253c85-6d6d-4152-99a6-e0b600b2e52c"). InnerVolumeSpecName "kube-api-access-n7n4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:27:28.945721 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.945697    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c253c85-6d6d-4152-99a6-e0b600b2e52c" (UID: "1c253c85-6d6d-4152-99a6-e0b600b2e52c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:27:28.956514 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:28.956486    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c253c85-6d6d-4152-99a6-e0b600b2e52c" (UID: "1c253c85-6d6d-4152-99a6-e0b600b2e52c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:29.013186 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.013154    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c253c85-6d6d-4152-99a6-e0b600b2e52c" (UID: "1c253c85-6d6d-4152-99a6-e0b600b2e52c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:29.043794 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.043756    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7n4k\" (UniqueName: \"kubernetes.io/projected/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kube-api-access-n7n4k\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:27:29.043794 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.043778    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:27:29.043794 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.043788    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:27:29.043794 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.043795    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c253c85-6d6d-4152-99a6-e0b600b2e52c-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:27:29.044124 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.043806    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:27:29.044124 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.043814    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c253c85-6d6d-4152-99a6-e0b600b2e52c-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:27:29.865519 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.865486    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"
Jul 09 18:27:29.885109 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.885075    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"]
Jul 09 18:27:29.888662 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:29.888634    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775b7467d7gkzsf"]
Jul 09 18:27:31.670626 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:31.670580    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" path="/var/lib/kubelet/pods/1c253c85-6d6d-4152-99a6-e0b600b2e52c/volumes"
Jul 09 18:27:40.795171 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.795120    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:27:40.795762 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.795740    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main"
Jul 09 18:27:40.795843 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.795765    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main"
Jul 09 18:27:40.795843 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.795782    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="storage-initializer"
Jul 09 18:27:40.795843 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.795789    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="storage-initializer"
Jul 09 18:27:40.796097 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.795872    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c253c85-6d6d-4152-99a6-e0b600b2e52c" containerName="main"
Jul 09 18:27:40.798241 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.798220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.801020 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.800997    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-5w6nc\""
Jul 09 18:27:40.801020 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.801005    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 09 18:27:40.812002 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.811979    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:27:40.847469 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.847443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.847590 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.847498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.847590 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.847579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzpzh\" (UniqueName: \"kubernetes.io/projected/4cbeff43-9161-4b45-95a9-d9b946232e14-kube-api-access-pzpzh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.847673 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.847626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.847673 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.847661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.847742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.847691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.847742 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.847725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbeff43-9161-4b45-95a9-d9b946232e14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.948890 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.948860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.948910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.948942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzpzh\" (UniqueName: \"kubernetes.io/projected/4cbeff43-9161-4b45-95a9-d9b946232e14-kube-api-access-pzpzh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.948970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.948992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.949015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.949066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbeff43-9161-4b45-95a9-d9b946232e14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.949391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949488 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.949459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949555 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.949467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.949645 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.949618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.951408 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.951384    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.951479 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.951463    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbeff43-9161-4b45-95a9-d9b946232e14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:40.957752 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:40.957729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzpzh\" (UniqueName: \"kubernetes.io/projected/4cbeff43-9161-4b45-95a9-d9b946232e14-kube-api-access-pzpzh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:41.108829 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:41.108795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:41.250388 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:41.250360    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:27:41.252401 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:27:41.252376    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbeff43_9161_4b45_95a9_d9b946232e14.slice/crio-afde07ac797f64505aa8672fe5a9d177a98d13798d047a5df3db555cb68e9553 WatchSource:0}: Error finding container afde07ac797f64505aa8672fe5a9d177a98d13798d047a5df3db555cb68e9553: Status 404 returned error can't find the container with id afde07ac797f64505aa8672fe5a9d177a98d13798d047a5df3db555cb68e9553
Jul 09 18:27:41.929835 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:41.929800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4cbeff43-9161-4b45-95a9-d9b946232e14","Type":"ContainerStarted","Data":"21e00a88121f276a74fcf011348bc8c0b54443b59184ba3dfd4771cabbdbdf69"}
Jul 09 18:27:41.930330 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:41.929862    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4cbeff43-9161-4b45-95a9-d9b946232e14","Type":"ContainerStarted","Data":"afde07ac797f64505aa8672fe5a9d177a98d13798d047a5df3db555cb68e9553"}
Jul 09 18:27:44.970826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:44.970722    2576 generic.go:356] "Generic (PLEG): container finished" podID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerID="21e00a88121f276a74fcf011348bc8c0b54443b59184ba3dfd4771cabbdbdf69" exitCode=0
Jul 09 18:27:44.970826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:44.970800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4cbeff43-9161-4b45-95a9-d9b946232e14","Type":"ContainerDied","Data":"21e00a88121f276a74fcf011348bc8c0b54443b59184ba3dfd4771cabbdbdf69"}
Jul 09 18:27:45.979545 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:45.979504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4cbeff43-9161-4b45-95a9-d9b946232e14","Type":"ContainerStarted","Data":"d0e1e4e7091561e8b48d801a190670be37a48f90b490e2a3f959854e1ab6b23e"}
Jul 09 18:27:46.007690 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:46.007643    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.614212368 podStartE2EDuration="6.007626903s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.393414535s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:46.004938914 +0000 UTC m=+2895.535715939" watchObservedRunningTime="2026-07-09 18:27:46.007626903 +0000 UTC m=+2895.538403932"
Jul 09 18:27:46.109779 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:46.109745    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:51.109718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:51.109685    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:27:51.111372 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:51.111344    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:27:57.553590 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:27:57.553557    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"]
Jul 09 18:28:01.109402 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:01.109364    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:28:11.109830 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:11.109791    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:28:12.577577 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:12.577525    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" containerID="cri-o://54850454fe2f93a5a8e086ed4f9a1e5a62b7717da4bab23f39adc5d765b3691c" gracePeriod=45
Jul 09 18:28:13.140317 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.140286    2576 generic.go:356] "Generic (PLEG): container finished" podID="dc6b214b-35d0-4388-8391-16551267b310" containerID="54850454fe2f93a5a8e086ed4f9a1e5a62b7717da4bab23f39adc5d765b3691c" exitCode=0
Jul 09 18:28:13.140506 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.140364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" event={"ID":"dc6b214b-35d0-4388-8391-16551267b310","Type":"ContainerDied","Data":"54850454fe2f93a5a8e086ed4f9a1e5a62b7717da4bab23f39adc5d765b3691c"}
Jul 09 18:28:13.257312 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.257292    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:28:13.373753 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373730    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-kserve-provision-location\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.373910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373772    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b214b-35d0-4388-8391-16551267b310-tls-certs\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.373910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373801    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dzln\" (UniqueName: \"kubernetes.io/projected/dc6b214b-35d0-4388-8391-16551267b310-kube-api-access-6dzln\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.373910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373815    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-home\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.373910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373854    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-model-cache\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.373910 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373882    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-dshm\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.374233 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373932    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-tmp-dir\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.374233 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.373986    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b214b-35d0-4388-8391-16551267b310-cabundle-cert\") pod \"dc6b214b-35d0-4388-8391-16551267b310\" (UID: \"dc6b214b-35d0-4388-8391-16551267b310\") "
Jul 09 18:28:13.374233 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.374190    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:13.374403 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.374275    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-home" (OuterVolumeSpecName: "home") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:13.374677 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.374656    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:13.374795 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.374731    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:13.374795 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.374674    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6b214b-35d0-4388-8391-16551267b310-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:13.376229 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.376204    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6b214b-35d0-4388-8391-16551267b310-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:13.377863 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.377831    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6b214b-35d0-4388-8391-16551267b310-kube-api-access-6dzln" (OuterVolumeSpecName: "kube-api-access-6dzln") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "kube-api-access-6dzln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:13.378152 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.378126    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-dshm" (OuterVolumeSpecName: "dshm") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:13.397318 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.397289    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:13.443922 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.443885    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc6b214b-35d0-4388-8391-16551267b310" (UID: "dc6b214b-35d0-4388-8391-16551267b310"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:13.476034 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.476004    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:13.476034 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.476028    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b214b-35d0-4388-8391-16551267b310-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:13.476225 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.476062    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzln\" (UniqueName: \"kubernetes.io/projected/dc6b214b-35d0-4388-8391-16551267b310-kube-api-access-6dzln\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:13.476225 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.476070    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:13.476225 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.476079    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b214b-35d0-4388-8391-16551267b310-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:13.476225 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:13.476087    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b214b-35d0-4388-8391-16551267b310-cabundle-cert\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:28:14.147227 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:14.147202    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"
Jul 09 18:28:14.147691 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:14.147234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" event={"ID":"dc6b214b-35d0-4388-8391-16551267b310","Type":"ContainerDied","Data":"9ba79fa0344c06c4b4687dfe3d091f6d703e44c6740ac36fa447452cf54cc13d"}
Jul 09 18:28:14.147691 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:14.147296    2576 scope.go:117] "RemoveContainer" containerID="54850454fe2f93a5a8e086ed4f9a1e5a62b7717da4bab23f39adc5d765b3691c"
Jul 09 18:28:14.157541 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:14.157509    2576 scope.go:117] "RemoveContainer" containerID="6875a68cc904e787916a55d441aeeaca10eaa35bffa1ab9d6eec3bf9866037f3"
Jul 09 18:28:14.169463 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:14.169441    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"]
Jul 09 18:28:14.173849 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:14.173826    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5"]
Jul 09 18:28:14.210551 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:14.210519    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-84645dc545-rjkj5" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:28:15.670507 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:15.670471    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6b214b-35d0-4388-8391-16551267b310" path="/var/lib/kubelet/pods/dc6b214b-35d0-4388-8391-16551267b310/volumes"
Jul 09 18:28:21.109613 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:21.109558    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:28:31.034638 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.034608    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"]
Jul 09 18:28:31.035101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.035012    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main"
Jul 09 18:28:31.035101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.035024    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main"
Jul 09 18:28:31.035101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.035053    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="storage-initializer"
Jul 09 18:28:31.035101 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.035063    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="storage-initializer"
Jul 09 18:28:31.035240 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.035141    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6b214b-35d0-4388-8391-16551267b310" containerName="main"
Jul 09 18:28:31.037195 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.037176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.040076 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.040053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 09 18:28:31.053334 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.053313    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"]
Jul 09 18:28:31.109204 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.109166    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:28:31.134652 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.134848 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-model-cache\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.134848 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7hm\" (UniqueName: \"kubernetes.io/projected/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kube-api-access-kv7hm\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.134848 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.134848 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-dshm\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.135021 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-home\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.135021 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tls-certs\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.135021 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.134975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.235610 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.235585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.235763 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.235644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.235763 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.235673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-model-cache\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.235763 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.235698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7hm\" (UniqueName: \"kubernetes.io/projected/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kube-api-access-kv7hm\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.235763 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.235732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.235958 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.235774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-dshm\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.236140 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.236110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-home\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.236219 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.236207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tls-certs\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.236435 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.236406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.236716 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.236669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.236716 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.236690    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-model-cache\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.236886 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.236833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.236950 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.236926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-home\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.238956 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.238938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-dshm\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.239137 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.239117    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tls-certs\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.245128 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.245103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7hm\" (UniqueName: \"kubernetes.io/projected/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kube-api-access-kv7hm\") pod \"lora-multiple-adapters-test-kserve-857c79454-z99g7\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.347133 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.347092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:31.500818 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:31.500793    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"]
Jul 09 18:28:31.503609 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:28:31.503583    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22eafda2_4dd0_4e9b_9e84_07ba6f9ac9ec.slice/crio-8af828b0d90432e5751f10266826a3065cbf663895b718b55a5b6dd1e9a10ba0 WatchSource:0}: Error finding container 8af828b0d90432e5751f10266826a3065cbf663895b718b55a5b6dd1e9a10ba0: Status 404 returned error can't find the container with id 8af828b0d90432e5751f10266826a3065cbf663895b718b55a5b6dd1e9a10ba0
Jul 09 18:28:32.241539 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:32.241506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" event={"ID":"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec","Type":"ContainerStarted","Data":"0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970"}
Jul 09 18:28:32.241918 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:32.241567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" event={"ID":"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec","Type":"ContainerStarted","Data":"8af828b0d90432e5751f10266826a3065cbf663895b718b55a5b6dd1e9a10ba0"}
Jul 09 18:28:36.272553 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:36.272519    2576 generic.go:356] "Generic (PLEG): container finished" podID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerID="0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970" exitCode=0
Jul 09 18:28:36.272938 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:36.272578    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" event={"ID":"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec","Type":"ContainerDied","Data":"0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970"}
Jul 09 18:28:37.280455 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:37.280419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" event={"ID":"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec","Type":"ContainerStarted","Data":"b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e"}
Jul 09 18:28:37.310650 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:37.310590    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podStartSLOduration=2.214926718 podStartE2EDuration="6.310576115s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.095649397s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:37.308636556 +0000 UTC m=+2946.839413585" watchObservedRunningTime="2026-07-09 18:28:37.310576115 +0000 UTC m=+2946.841353145"
Jul 09 18:28:37.347950 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:37.347921    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:41.110340 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:41.110278    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:28:41.348338 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:41.348299    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:28:41.349868 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:41.349836    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:28:51.109994 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:51.109950    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:28:51.347871 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:28:51.347824    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:29:01.109946 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:01.109856    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:29:01.347871 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:01.347825    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:29:11.109430 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:11.109383    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:29:11.347903 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:11.347852    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:29:21.119031 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:21.119002    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:29:21.126631 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:21.126606    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:29:21.348216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:21.348171    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:29:31.213464 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:31.213437    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:29:31.221629 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:31.221607    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:29:31.347588 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:31.347539    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:29:41.348364 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:41.348294    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:29:51.358032 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:51.358001    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:29:51.365735 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:29:51.365706    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:30:40.154139 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:40.154107    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"]
Jul 09 18:30:46.430372 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:46.430325    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:30:55.181752 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.181687    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" containerID="cri-o://b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e" gracePeriod=45
Jul 09 18:30:55.359462 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.359357    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 09 18:30:55.849896 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.849864    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:30:55.981707 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981613    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv7hm\" (UniqueName: \"kubernetes.io/projected/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kube-api-access-kv7hm\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.981707 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981700    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-model-cache\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.981945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981722    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kserve-provision-location\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.981945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981745    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-cabundle-cert\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.981945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981784    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-home\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.981945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981827    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-dshm\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.981945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tmp-dir\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.981945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.981869    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tls-certs\") pod \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\" (UID: \"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec\") "
Jul 09 18:30:55.982362 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.982018    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:55.982362 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.982173    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:30:55.982362 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.982263    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:55.982362 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.982280    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-home" (OuterVolumeSpecName: "home") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:55.984099 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.984075    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:30:55.984219 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.984189    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-dshm" (OuterVolumeSpecName: "dshm") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:55.984391 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.984357    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kube-api-access-kv7hm" (OuterVolumeSpecName: "kube-api-access-kv7hm") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "kube-api-access-kv7hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:30:55.994750 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:55.994712    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:56.054827 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.054770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" (UID: "22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:56.074554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.074520    2576 generic.go:356] "Generic (PLEG): container finished" podID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerID="b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e" exitCode=0
Jul 09 18:30:56.074718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.074570    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"
Jul 09 18:30:56.074718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.074643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" event={"ID":"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec","Type":"ContainerDied","Data":"b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e"}
Jul 09 18:30:56.074718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.074694    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7" event={"ID":"22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec","Type":"ContainerDied","Data":"8af828b0d90432e5751f10266826a3065cbf663895b718b55a5b6dd1e9a10ba0"}
Jul 09 18:30:56.074718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.074717    2576 scope.go:117] "RemoveContainer" containerID="b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e"
Jul 09 18:30:56.083196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.082961    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7hm\" (UniqueName: \"kubernetes.io/projected/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kube-api-access-kv7hm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:56.083196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.082983    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:56.083196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.082993    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-cabundle-cert\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:56.083196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.083002    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:56.083196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.083010    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:56.083196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.083052    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:56.083196 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.083063    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:30:56.084809 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.084790    2576 scope.go:117] "RemoveContainer" containerID="0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970"
Jul 09 18:30:56.093127 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.093108    2576 scope.go:117] "RemoveContainer" containerID="b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e"
Jul 09 18:30:56.093507 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:30:56.093488    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e\": container with ID starting with b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e not found: ID does not exist" containerID="b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e"
Jul 09 18:30:56.093589 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.093518    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e"} err="failed to get container status \"b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e\": rpc error: code = NotFound desc = could not find container \"b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e\": container with ID starting with b3f789664fd1817f2a3676603adf6a0247bcadbebb027938af14d7d1c74c056e not found: ID does not exist"
Jul 09 18:30:56.093635 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.093596    2576 scope.go:117] "RemoveContainer" containerID="0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970"
Jul 09 18:30:56.093879 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:30:56.093858    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970\": container with ID starting with 0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970 not found: ID does not exist" containerID="0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970"
Jul 09 18:30:56.093942 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.093884    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970"} err="failed to get container status \"0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970\": rpc error: code = NotFound desc = could not find container \"0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970\": container with ID starting with 0ad81cad9a8f25fee7d13d9b4468949b691206ce40e16b3e2f399fb737d55970 not found: ID does not exist"
Jul 09 18:30:56.099108 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.099077    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"]
Jul 09 18:30:56.105343 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:56.105320    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-857c79454-z99g7"]
Jul 09 18:30:57.669625 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:30:57.669589    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" path="/var/lib/kubelet/pods/22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec/volumes"
Jul 09 18:31:01.454566 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:01.454512    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" containerID="cri-o://d0e1e4e7091561e8b48d801a190670be37a48f90b490e2a3f959854e1ab6b23e" gracePeriod=45
Jul 09 18:31:02.109671 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.109634    2576 generic.go:356] "Generic (PLEG): container finished" podID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerID="d0e1e4e7091561e8b48d801a190670be37a48f90b490e2a3f959854e1ab6b23e" exitCode=0
Jul 09 18:31:02.109839 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.109710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4cbeff43-9161-4b45-95a9-d9b946232e14","Type":"ContainerDied","Data":"d0e1e4e7091561e8b48d801a190670be37a48f90b490e2a3f959854e1ab6b23e"}
Jul 09 18:31:02.120082 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.120016    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 18:31:02.225363 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.225335    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:31:02.351860 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.351812    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-dshm\") pod \"4cbeff43-9161-4b45-95a9-d9b946232e14\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") "
Jul 09 18:31:02.352077 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.351886    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbeff43-9161-4b45-95a9-d9b946232e14-tls-certs\") pod \"4cbeff43-9161-4b45-95a9-d9b946232e14\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") "
Jul 09 18:31:02.352077 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.351917    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzpzh\" (UniqueName: \"kubernetes.io/projected/4cbeff43-9161-4b45-95a9-d9b946232e14-kube-api-access-pzpzh\") pod \"4cbeff43-9161-4b45-95a9-d9b946232e14\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") "
Jul 09 18:31:02.352077 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.351969    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-model-cache\") pod \"4cbeff43-9161-4b45-95a9-d9b946232e14\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") "
Jul 09 18:31:02.352077 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.352065    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-tmp-dir\") pod \"4cbeff43-9161-4b45-95a9-d9b946232e14\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") "
Jul 09 18:31:02.352266 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.352105    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-kserve-provision-location\") pod \"4cbeff43-9161-4b45-95a9-d9b946232e14\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") "
Jul 09 18:31:02.352266 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.352176    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-home\") pod \"4cbeff43-9161-4b45-95a9-d9b946232e14\" (UID: \"4cbeff43-9161-4b45-95a9-d9b946232e14\") "
Jul 09 18:31:02.352378 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.352343    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cbeff43-9161-4b45-95a9-d9b946232e14" (UID: "4cbeff43-9161-4b45-95a9-d9b946232e14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:02.353009 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.352980    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-home" (OuterVolumeSpecName: "home") pod "4cbeff43-9161-4b45-95a9-d9b946232e14" (UID: "4cbeff43-9161-4b45-95a9-d9b946232e14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:02.353220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.353200    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:31:02.353287 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.353225    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:31:02.354110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.354076    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbeff43-9161-4b45-95a9-d9b946232e14-kube-api-access-pzpzh" (OuterVolumeSpecName: "kube-api-access-pzpzh") pod "4cbeff43-9161-4b45-95a9-d9b946232e14" (UID: "4cbeff43-9161-4b45-95a9-d9b946232e14"). InnerVolumeSpecName "kube-api-access-pzpzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:31:02.354215 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.354136    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-dshm" (OuterVolumeSpecName: "dshm") pod "4cbeff43-9161-4b45-95a9-d9b946232e14" (UID: "4cbeff43-9161-4b45-95a9-d9b946232e14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:02.354262 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.354235    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbeff43-9161-4b45-95a9-d9b946232e14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cbeff43-9161-4b45-95a9-d9b946232e14" (UID: "4cbeff43-9161-4b45-95a9-d9b946232e14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:31:02.366414 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.366381    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cbeff43-9161-4b45-95a9-d9b946232e14" (UID: "4cbeff43-9161-4b45-95a9-d9b946232e14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:02.422151 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.422114    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cbeff43-9161-4b45-95a9-d9b946232e14" (UID: "4cbeff43-9161-4b45-95a9-d9b946232e14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:02.453950 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.453914    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:31:02.453950 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.453943    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:31:02.453950 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.453954    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbeff43-9161-4b45-95a9-d9b946232e14-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:31:02.454216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.453962    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbeff43-9161-4b45-95a9-d9b946232e14-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:31:02.454216 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:02.453971    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzpzh\" (UniqueName: \"kubernetes.io/projected/4cbeff43-9161-4b45-95a9-d9b946232e14-kube-api-access-pzpzh\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:31:03.116628 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:03.116599    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:31:03.117298 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:03.116597    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4cbeff43-9161-4b45-95a9-d9b946232e14","Type":"ContainerDied","Data":"afde07ac797f64505aa8672fe5a9d177a98d13798d047a5df3db555cb68e9553"}
Jul 09 18:31:03.117298 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:03.116732    2576 scope.go:117] "RemoveContainer" containerID="d0e1e4e7091561e8b48d801a190670be37a48f90b490e2a3f959854e1ab6b23e"
Jul 09 18:31:03.126227 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:03.126205    2576 scope.go:117] "RemoveContainer" containerID="21e00a88121f276a74fcf011348bc8c0b54443b59184ba3dfd4771cabbdbdf69"
Jul 09 18:31:03.147547 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:03.147523    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:31:03.153734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:03.153710    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:31:03.669512 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:03.669469    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" path="/var/lib/kubelet/pods/4cbeff43-9161-4b45-95a9-d9b946232e14/volumes"
Jul 09 18:31:07.669170 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669140    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"]
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669481    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main"
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669492    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main"
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669512    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="storage-initializer"
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669518    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="storage-initializer"
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669524    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main"
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669530    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main"
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669540    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="storage-initializer"
Jul 09 18:31:07.669554 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669546    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="storage-initializer"
Jul 09 18:31:07.669832 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669605    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbeff43-9161-4b45-95a9-d9b946232e14" containerName="main"
Jul 09 18:31:07.669832 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.669616    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22eafda2-4dd0-4e9b-9e84-07ba6f9ac9ec" containerName="main"
Jul 09 18:31:07.674409 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.674386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.677211 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.677176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-2jv28\""
Jul 09 18:31:07.678505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.678484    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 09 18:31:07.681134 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.681108    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"]
Jul 09 18:31:07.805242 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.805197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.805451 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.805262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dlc\" (UniqueName: \"kubernetes.io/projected/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kube-api-access-q7dlc\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.805451 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.805377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.805451 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.805441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.906652 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.906611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.906844 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.906677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dlc\" (UniqueName: \"kubernetes.io/projected/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kube-api-access-q7dlc\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.906844 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.906720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.906844 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.906750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.907158 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.907136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.907158 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.907156    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.909207 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.909183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.916392 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.916368    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dlc\" (UniqueName: \"kubernetes.io/projected/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kube-api-access-q7dlc\") pod \"tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:07.986238 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:07.986140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:08.133315 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:08.130846    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"]
Jul 09 18:31:08.135261 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:31:08.135231    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4dd276_d7ef_4bdb_9959_26c33bf61a0d.slice/crio-e0d7f3cd3820df4b737ffe5f84474921cd7ec9e183d61dc13d6e57279587c973 WatchSource:0}: Error finding container e0d7f3cd3820df4b737ffe5f84474921cd7ec9e183d61dc13d6e57279587c973: Status 404 returned error can't find the container with id e0d7f3cd3820df4b737ffe5f84474921cd7ec9e183d61dc13d6e57279587c973
Jul 09 18:31:08.151904 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:08.151878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" event={"ID":"da4dd276-d7ef-4bdb-9959-26c33bf61a0d","Type":"ContainerStarted","Data":"e0d7f3cd3820df4b737ffe5f84474921cd7ec9e183d61dc13d6e57279587c973"}
Jul 09 18:31:09.158459 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:09.158372    2576 generic.go:356] "Generic (PLEG): container finished" podID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerID="f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd" exitCode=0
Jul 09 18:31:09.158459 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:09.158428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" event={"ID":"da4dd276-d7ef-4bdb-9959-26c33bf61a0d","Type":"ContainerDied","Data":"f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd"}
Jul 09 18:31:10.166025 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:10.165994    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" event={"ID":"da4dd276-d7ef-4bdb-9959-26c33bf61a0d","Type":"ContainerStarted","Data":"a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d"}
Jul 09 18:31:10.166453 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:10.166132    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:31:10.192022 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:10.191967    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" podStartSLOduration=2.536737872 podStartE2EDuration="3.191947393s" totalImagesPullingTime="0s" totalInitContainerRuntime="655.209521ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:10.188945099 +0000 UTC m=+3099.719722140" watchObservedRunningTime="2026-07-09 18:31:10.191947393 +0000 UTC m=+3099.722724423"
Jul 09 18:31:31.408810 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:31.408771    2576 scope.go:117] "RemoveContainer" containerID="d0c50c6f4e29047f245b50edd5576ad172d3e3db8554a65388a0702e9510dd30"
Jul 09 18:31:31.417300 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:31.417275    2576 scope.go:117] "RemoveContainer" containerID="8377474bfbceb266b4ce095bc7857b1472cc440689cd3670dfd593694b86d81e"
Jul 09 18:31:41.173558 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:31:41.173529    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:33:12.930357 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:12.930326    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"]
Jul 09 18:33:12.939210 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:33:12.939177    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:33:12.939210 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:33:12.939210 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:33:12.939210 ip-10-0-132-16 kubenswrapper[2576]:  > containerID="a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d" cmd=["/bin/sleep","15"]
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:12.939223    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" message="" err=<
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]:  >
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:33:12.939241    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerName="main" containerID="cri-o://a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d"
Jul 09 18:33:12.939456 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:12.939283    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerName="main" containerID="cri-o://a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d" gracePeriod=60
Jul 09 18:33:13.204333 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.204308    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:33:13.347966 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.347922    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kserve-provision-location\") pod \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") "
Jul 09 18:33:13.348159 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.348020    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tls-certs\") pod \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") "
Jul 09 18:33:13.348159 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.348104    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tokenizer-uds\") pod \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") "
Jul 09 18:33:13.348159 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.348136    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7dlc\" (UniqueName: \"kubernetes.io/projected/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kube-api-access-q7dlc\") pod \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\" (UID: \"da4dd276-d7ef-4bdb-9959-26c33bf61a0d\") "
Jul 09 18:33:13.348398 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.348371    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "da4dd276-d7ef-4bdb-9959-26c33bf61a0d" (UID: "da4dd276-d7ef-4bdb-9959-26c33bf61a0d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:13.348505 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.348486    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tokenizer-uds\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:33:13.348740 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.348722    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da4dd276-d7ef-4bdb-9959-26c33bf61a0d" (UID: "da4dd276-d7ef-4bdb-9959-26c33bf61a0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:13.350028 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.350005    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da4dd276-d7ef-4bdb-9959-26c33bf61a0d" (UID: "da4dd276-d7ef-4bdb-9959-26c33bf61a0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:33:13.350220 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.350202    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kube-api-access-q7dlc" (OuterVolumeSpecName: "kube-api-access-q7dlc") pod "da4dd276-d7ef-4bdb-9959-26c33bf61a0d" (UID: "da4dd276-d7ef-4bdb-9959-26c33bf61a0d"). InnerVolumeSpecName "kube-api-access-q7dlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:33:13.449738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.449679    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:33:13.449738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.449700    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:33:13.449738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.449710    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dlc\" (UniqueName: \"kubernetes.io/projected/da4dd276-d7ef-4bdb-9959-26c33bf61a0d-kube-api-access-q7dlc\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:33:13.783468 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.783367    2576 generic.go:356] "Generic (PLEG): container finished" podID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerID="a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d" exitCode=0
Jul 09 18:33:13.783468 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.783406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" event={"ID":"da4dd276-d7ef-4bdb-9959-26c33bf61a0d","Type":"ContainerDied","Data":"a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d"}
Jul 09 18:33:13.783468 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.783428    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"
Jul 09 18:33:13.783468 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.783468    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t" event={"ID":"da4dd276-d7ef-4bdb-9959-26c33bf61a0d","Type":"ContainerDied","Data":"e0d7f3cd3820df4b737ffe5f84474921cd7ec9e183d61dc13d6e57279587c973"}
Jul 09 18:33:13.783783 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.783499    2576 scope.go:117] "RemoveContainer" containerID="a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d"
Jul 09 18:33:13.792333 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.792316    2576 scope.go:117] "RemoveContainer" containerID="f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd"
Jul 09 18:33:13.799744 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.799725    2576 scope.go:117] "RemoveContainer" containerID="a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d"
Jul 09 18:33:13.800214 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:33:13.800139    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d\": container with ID starting with a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d not found: ID does not exist" containerID="a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d"
Jul 09 18:33:13.800214 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.800186    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d"} err="failed to get container status \"a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d\": rpc error: code = NotFound desc = could not find container \"a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d\": container with ID starting with a8a6401bb73784eee29b6e89a4ea4168a5737254b70f013ed31adb6534f2326d not found: ID does not exist"
Jul 09 18:33:13.800501 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.800207    2576 scope.go:117] "RemoveContainer" containerID="f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd"
Jul 09 18:33:13.800569 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:33:13.800547    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd\": container with ID starting with f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd not found: ID does not exist" containerID="f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd"
Jul 09 18:33:13.800631 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.800578    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd"} err="failed to get container status \"f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd\": rpc error: code = NotFound desc = could not find container \"f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd\": container with ID starting with f91679320e3e3f80a4e80ef663289a1e1f439ac51c459f693229f374b28f9ecd not found: ID does not exist"
Jul 09 18:33:13.803804 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.802821    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"]
Jul 09 18:33:13.807919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:13.807892    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-655f6fd4c7-4557t"]
Jul 09 18:33:15.670289 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:15.670247    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" path="/var/lib/kubelet/pods/da4dd276-d7ef-4bdb-9959-26c33bf61a0d/volumes"
Jul 09 18:33:36.545787 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.545757    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"]
Jul 09 18:33:36.546387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.546315    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerName="storage-initializer"
Jul 09 18:33:36.546387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.546336    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerName="storage-initializer"
Jul 09 18:33:36.546387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.546375    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerName="main"
Jul 09 18:33:36.546387 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.546383    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerName="main"
Jul 09 18:33:36.546614 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.546479    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4dd276-d7ef-4bdb-9959-26c33bf61a0d" containerName="main"
Jul 09 18:33:36.551023 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.551001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.555607 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.555581    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-xf8t4\""
Jul 09 18:33:36.555607 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.555596    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 09 18:33:36.563521 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.563491    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"]
Jul 09 18:33:36.633334 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.633309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvtp\" (UniqueName: \"kubernetes.io/projected/446d932c-6477-40a3-a93d-467a553111f4-kube-api-access-kzvtp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.633466 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.633355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/446d932c-6477-40a3-a93d-467a553111f4-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.633466 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.633395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446d932c-6477-40a3-a93d-467a553111f4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.734572 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.734542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvtp\" (UniqueName: \"kubernetes.io/projected/446d932c-6477-40a3-a93d-467a553111f4-kube-api-access-kzvtp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.734695 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.734577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/446d932c-6477-40a3-a93d-467a553111f4-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.734695 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.734594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446d932c-6477-40a3-a93d-467a553111f4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.734956 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.734933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/446d932c-6477-40a3-a93d-467a553111f4-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.736938 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.736921    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446d932c-6477-40a3-a93d-467a553111f4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.743168 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.743145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvtp\" (UniqueName: \"kubernetes.io/projected/446d932c-6477-40a3-a93d-467a553111f4-kube-api-access-kzvtp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.862712 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.862688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:36.998284 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:36.998258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"]
Jul 09 18:33:37.000275 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:33:37.000251    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446d932c_6477_40a3_a93d_467a553111f4.slice/crio-e3c127079bbdfe820f386a39cc35d0939a6a0d3ba24c5f8c1b2a110cc1ea9b1a WatchSource:0}: Error finding container e3c127079bbdfe820f386a39cc35d0939a6a0d3ba24c5f8c1b2a110cc1ea9b1a: Status 404 returned error can't find the container with id e3c127079bbdfe820f386a39cc35d0939a6a0d3ba24c5f8c1b2a110cc1ea9b1a
Jul 09 18:33:37.898258 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:37.898215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" event={"ID":"446d932c-6477-40a3-a93d-467a553111f4","Type":"ContainerStarted","Data":"ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876"}
Jul 09 18:33:37.898607 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:37.898277    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" event={"ID":"446d932c-6477-40a3-a93d-467a553111f4","Type":"ContainerStarted","Data":"e3c127079bbdfe820f386a39cc35d0939a6a0d3ba24c5f8c1b2a110cc1ea9b1a"}
Jul 09 18:33:37.898607 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:37.898315    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:33:37.918733 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:37.918685    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" podStartSLOduration=1.918669511 podStartE2EDuration="1.918669511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:37.916937425 +0000 UTC m=+3247.447714456" watchObservedRunningTime="2026-07-09 18:33:37.918669511 +0000 UTC m=+3247.449446541"
Jul 09 18:33:49.178335 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.178301    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"]
Jul 09 18:33:49.181976 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.181958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.185060 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.185012    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-rvhb4\""
Jul 09 18:33:49.185170 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.185012    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 09 18:33:49.195385 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.195367    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"]
Jul 09 18:33:49.241341 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.241315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj462\" (UniqueName: \"kubernetes.io/projected/ce0621b6-098f-4596-94f6-76339db6efb5-kube-api-access-pj462\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.241447 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.241348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.241447 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.241376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.241539 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.241495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce0621b6-098f-4596-94f6-76339db6efb5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.342849 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.342824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.342948 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.342854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.342948 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.342920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce0621b6-098f-4596-94f6-76339db6efb5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.343091 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.342953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj462\" (UniqueName: \"kubernetes.io/projected/ce0621b6-098f-4596-94f6-76339db6efb5-kube-api-access-pj462\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.343185 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.343165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.343286 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.343266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.345274 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.345258    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce0621b6-098f-4596-94f6-76339db6efb5-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.351110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.351092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj462\" (UniqueName: \"kubernetes.io/projected/ce0621b6-098f-4596-94f6-76339db6efb5-kube-api-access-pj462\") pod \"prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.491604 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.491544    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:33:49.638787 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.638761    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"]
Jul 09 18:33:49.640731 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:33:49.640706    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0621b6_098f_4596_94f6_76339db6efb5.slice/crio-53e8596af74635df2111d35a61317e3542036d594bc53f6c7915b8aba2d6f913 WatchSource:0}: Error finding container 53e8596af74635df2111d35a61317e3542036d594bc53f6c7915b8aba2d6f913: Status 404 returned error can't find the container with id 53e8596af74635df2111d35a61317e3542036d594bc53f6c7915b8aba2d6f913
Jul 09 18:33:49.956349 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.956314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" event={"ID":"ce0621b6-098f-4596-94f6-76339db6efb5","Type":"ContainerStarted","Data":"c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08"}
Jul 09 18:33:49.956549 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:49.956361    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" event={"ID":"ce0621b6-098f-4596-94f6-76339db6efb5","Type":"ContainerStarted","Data":"53e8596af74635df2111d35a61317e3542036d594bc53f6c7915b8aba2d6f913"}
Jul 09 18:33:50.962880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:50.962850    2576 generic.go:356] "Generic (PLEG): container finished" podID="ce0621b6-098f-4596-94f6-76339db6efb5" containerID="c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08" exitCode=0
Jul 09 18:33:50.963270 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:50.962943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" event={"ID":"ce0621b6-098f-4596-94f6-76339db6efb5","Type":"ContainerDied","Data":"c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08"}
Jul 09 18:33:51.969022 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:51.968992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" event={"ID":"ce0621b6-098f-4596-94f6-76339db6efb5","Type":"ContainerStarted","Data":"43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3"}
Jul 09 18:33:51.969406 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:33:51.969139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:34:08.904998 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:08.904966    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:34:08.935839 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:08.935794    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" podStartSLOduration=19.267889256 podStartE2EDuration="19.935779844s" totalImagesPullingTime="0s" totalInitContainerRuntime="667.890588ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:51.991721153 +0000 UTC m=+3261.522498202" watchObservedRunningTime="2026-07-09 18:34:08.935779844 +0000 UTC m=+3278.466556874"
Jul 09 18:34:10.732110 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:10.732076    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"]
Jul 09 18:34:10.741551 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:34:10.741512    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:34:10.741551 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:34:10.741551 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:34:10.741551 ip-10-0-132-16 kubenswrapper[2576]:  > containerID="ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876" cmd=["/bin/sleep","15"]
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:10.741570    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" message="" err=<
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]:  >
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:34:10.741587    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" podUID="446d932c-6477-40a3-a93d-467a553111f4" containerName="main" containerID="cri-o://ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876"
Jul 09 18:34:10.741869 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:10.741645    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" podUID="446d932c-6477-40a3-a93d-467a553111f4" containerName="main" containerID="cri-o://ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876" gracePeriod=60
Jul 09 18:34:10.994502 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:10.994449    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:34:11.058660 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.058627    2576 generic.go:356] "Generic (PLEG): container finished" podID="446d932c-6477-40a3-a93d-467a553111f4" containerID="ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876" exitCode=0
Jul 09 18:34:11.058660 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.058661    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"
Jul 09 18:34:11.058849 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.058705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" event={"ID":"446d932c-6477-40a3-a93d-467a553111f4","Type":"ContainerDied","Data":"ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876"}
Jul 09 18:34:11.058849 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.058762    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76" event={"ID":"446d932c-6477-40a3-a93d-467a553111f4","Type":"ContainerDied","Data":"e3c127079bbdfe820f386a39cc35d0939a6a0d3ba24c5f8c1b2a110cc1ea9b1a"}
Jul 09 18:34:11.058849 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.058786    2576 scope.go:117] "RemoveContainer" containerID="ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876"
Jul 09 18:34:11.067021 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.067005    2576 scope.go:117] "RemoveContainer" containerID="ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876"
Jul 09 18:34:11.067292 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:34:11.067276    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876\": container with ID starting with ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876 not found: ID does not exist" containerID="ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876"
Jul 09 18:34:11.067334 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.067300    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876"} err="failed to get container status \"ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876\": rpc error: code = NotFound desc = could not find container \"ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876\": container with ID starting with ec34b1e564253cf1ff5ec8f086a3d29fb95aa23819186f4b2ff9e456f25a3876 not found: ID does not exist"
Jul 09 18:34:11.134659 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.134637    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/446d932c-6477-40a3-a93d-467a553111f4-tokenizer-uds\") pod \"446d932c-6477-40a3-a93d-467a553111f4\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") "
Jul 09 18:34:11.134764 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.134684    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446d932c-6477-40a3-a93d-467a553111f4-tls-certs\") pod \"446d932c-6477-40a3-a93d-467a553111f4\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") "
Jul 09 18:34:11.134764 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.134731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzvtp\" (UniqueName: \"kubernetes.io/projected/446d932c-6477-40a3-a93d-467a553111f4-kube-api-access-kzvtp\") pod \"446d932c-6477-40a3-a93d-467a553111f4\" (UID: \"446d932c-6477-40a3-a93d-467a553111f4\") "
Jul 09 18:34:11.134882 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.134863    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446d932c-6477-40a3-a93d-467a553111f4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "446d932c-6477-40a3-a93d-467a553111f4" (UID: "446d932c-6477-40a3-a93d-467a553111f4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:11.136751 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.136728    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446d932c-6477-40a3-a93d-467a553111f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "446d932c-6477-40a3-a93d-467a553111f4" (UID: "446d932c-6477-40a3-a93d-467a553111f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:34:11.136843 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.136770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446d932c-6477-40a3-a93d-467a553111f4-kube-api-access-kzvtp" (OuterVolumeSpecName: "kube-api-access-kzvtp") pod "446d932c-6477-40a3-a93d-467a553111f4" (UID: "446d932c-6477-40a3-a93d-467a553111f4"). InnerVolumeSpecName "kube-api-access-kzvtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:11.235718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.235694    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvtp\" (UniqueName: \"kubernetes.io/projected/446d932c-6477-40a3-a93d-467a553111f4-kube-api-access-kzvtp\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:34:11.235826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.235720    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/446d932c-6477-40a3-a93d-467a553111f4-tokenizer-uds\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:34:11.235826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.235734    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446d932c-6477-40a3-a93d-467a553111f4-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:34:11.381232 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.381208    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"]
Jul 09 18:34:11.391482 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.391455    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f879pk76"]
Jul 09 18:34:11.670751 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:11.670676    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="446d932c-6477-40a3-a93d-467a553111f4" path="/var/lib/kubelet/pods/446d932c-6477-40a3-a93d-467a553111f4/volumes"
Jul 09 18:34:22.976085 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:22.976056    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:34:31.247767 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:31.247736    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:34:31.263034 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:31.263012    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:34:54.636123 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.636053    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"]
Jul 09 18:34:54.636542 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.636426    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446d932c-6477-40a3-a93d-467a553111f4" containerName="main"
Jul 09 18:34:54.636542 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.636436    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="446d932c-6477-40a3-a93d-467a553111f4" containerName="main"
Jul 09 18:34:54.636542 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.636492    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="446d932c-6477-40a3-a93d-467a553111f4" containerName="main"
Jul 09 18:34:54.639543 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.639522    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.643623 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.643606    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-cpzlx\""
Jul 09 18:34:54.643712 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.643662    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 09 18:34:54.650579 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.650559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"]
Jul 09 18:34:54.795762 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.795725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2vf\" (UniqueName: \"kubernetes.io/projected/8c4843da-2662-48ba-a82a-7c72e6e7816c-kube-api-access-bl2vf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.795892 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.795827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c4843da-2662-48ba-a82a-7c72e6e7816c-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.795892 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.795870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4843da-2662-48ba-a82a-7c72e6e7816c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.896582 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.896509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2vf\" (UniqueName: \"kubernetes.io/projected/8c4843da-2662-48ba-a82a-7c72e6e7816c-kube-api-access-bl2vf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.896582 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.896578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c4843da-2662-48ba-a82a-7c72e6e7816c-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.896735 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.896602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4843da-2662-48ba-a82a-7c72e6e7816c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.897003 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.896978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c4843da-2662-48ba-a82a-7c72e6e7816c-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.898976 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.898959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4843da-2662-48ba-a82a-7c72e6e7816c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.904654 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.904637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2vf\" (UniqueName: \"kubernetes.io/projected/8c4843da-2662-48ba-a82a-7c72e6e7816c-kube-api-access-bl2vf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:54.949262 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:54.949236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:55.090116 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:55.090082    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"]
Jul 09 18:34:55.090739 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:34:55.090716    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4843da_2662_48ba_a82a_7c72e6e7816c.slice/crio-7a2157a36b5050557d7b430f7786a5b4277f1253b140e88d89682e7f51f89b48 WatchSource:0}: Error finding container 7a2157a36b5050557d7b430f7786a5b4277f1253b140e88d89682e7f51f89b48: Status 404 returned error can't find the container with id 7a2157a36b5050557d7b430f7786a5b4277f1253b140e88d89682e7f51f89b48
Jul 09 18:34:55.276171 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:55.276087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" event={"ID":"8c4843da-2662-48ba-a82a-7c72e6e7816c","Type":"ContainerStarted","Data":"bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499"}
Jul 09 18:34:55.276171 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:55.276145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" event={"ID":"8c4843da-2662-48ba-a82a-7c72e6e7816c","Type":"ContainerStarted","Data":"7a2157a36b5050557d7b430f7786a5b4277f1253b140e88d89682e7f51f89b48"}
Jul 09 18:34:55.276379 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:55.276204    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:34:55.298401 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:34:55.298352    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" podStartSLOduration=1.29833919 podStartE2EDuration="1.29833919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:55.296710593 +0000 UTC m=+3324.827487624" watchObservedRunningTime="2026-07-09 18:34:55.29833919 +0000 UTC m=+3324.829116219"
Jul 09 18:35:26.282541 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:26.282515    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:35:27.580847 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.580818    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"]
Jul 09 18:35:27.590948 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:35:27.590918    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:35:27.590948 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:35:27.590948 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:35:27.590948 ip-10-0-132-16 kubenswrapper[2576]:  > containerID="bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499" cmd=["/bin/sleep","15"]
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.590961    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" message="" err=<
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]:  >
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:35:27.590974    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" podUID="8c4843da-2662-48ba-a82a-7c72e6e7816c" containerName="main" containerID="cri-o://bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499"
Jul 09 18:35:27.591232 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.591020    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" podUID="8c4843da-2662-48ba-a82a-7c72e6e7816c" containerName="main" containerID="cri-o://bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499" gracePeriod=60
Jul 09 18:35:27.835761 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.835701    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:35:27.971996 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.971972    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c4843da-2662-48ba-a82a-7c72e6e7816c-tokenizer-uds\") pod \"8c4843da-2662-48ba-a82a-7c72e6e7816c\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") "
Jul 09 18:35:27.972156 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.972067    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4843da-2662-48ba-a82a-7c72e6e7816c-tls-certs\") pod \"8c4843da-2662-48ba-a82a-7c72e6e7816c\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") "
Jul 09 18:35:27.972156 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.972099    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl2vf\" (UniqueName: \"kubernetes.io/projected/8c4843da-2662-48ba-a82a-7c72e6e7816c-kube-api-access-bl2vf\") pod \"8c4843da-2662-48ba-a82a-7c72e6e7816c\" (UID: \"8c4843da-2662-48ba-a82a-7c72e6e7816c\") "
Jul 09 18:35:27.972257 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.972233    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4843da-2662-48ba-a82a-7c72e6e7816c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8c4843da-2662-48ba-a82a-7c72e6e7816c" (UID: "8c4843da-2662-48ba-a82a-7c72e6e7816c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:35:27.972365 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.972349    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8c4843da-2662-48ba-a82a-7c72e6e7816c-tokenizer-uds\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:35:27.974132 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.974102    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4843da-2662-48ba-a82a-7c72e6e7816c-kube-api-access-bl2vf" (OuterVolumeSpecName: "kube-api-access-bl2vf") pod "8c4843da-2662-48ba-a82a-7c72e6e7816c" (UID: "8c4843da-2662-48ba-a82a-7c72e6e7816c"). InnerVolumeSpecName "kube-api-access-bl2vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:35:27.974317 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:27.974147    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4843da-2662-48ba-a82a-7c72e6e7816c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c4843da-2662-48ba-a82a-7c72e6e7816c" (UID: "8c4843da-2662-48ba-a82a-7c72e6e7816c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:35:28.072958 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.072921    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4843da-2662-48ba-a82a-7c72e6e7816c-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:35:28.072958 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.072957    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2vf\" (UniqueName: \"kubernetes.io/projected/8c4843da-2662-48ba-a82a-7c72e6e7816c-kube-api-access-bl2vf\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:35:28.436781 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.436749    2576 generic.go:356] "Generic (PLEG): container finished" podID="8c4843da-2662-48ba-a82a-7c72e6e7816c" containerID="bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499" exitCode=0
Jul 09 18:35:28.436954 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.436795    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"
Jul 09 18:35:28.436954 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.436839    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" event={"ID":"8c4843da-2662-48ba-a82a-7c72e6e7816c","Type":"ContainerDied","Data":"bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499"}
Jul 09 18:35:28.436954 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.436900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n" event={"ID":"8c4843da-2662-48ba-a82a-7c72e6e7816c","Type":"ContainerDied","Data":"7a2157a36b5050557d7b430f7786a5b4277f1253b140e88d89682e7f51f89b48"}
Jul 09 18:35:28.436954 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.436926    2576 scope.go:117] "RemoveContainer" containerID="bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499"
Jul 09 18:35:28.449166 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.449144    2576 scope.go:117] "RemoveContainer" containerID="bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499"
Jul 09 18:35:28.449785 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:35:28.449768    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499\": container with ID starting with bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499 not found: ID does not exist" containerID="bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499"
Jul 09 18:35:28.449837 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.449792    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499"} err="failed to get container status \"bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499\": rpc error: code = NotFound desc = could not find container \"bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499\": container with ID starting with bc23725dbf91b39c90dcf38d4a3f7e77a775dc4abee5daddc814ac7e96568499 not found: ID does not exist"
Jul 09 18:35:28.463713 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.463681    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"]
Jul 09 18:35:28.468545 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:28.468526    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9h92n"]
Jul 09 18:35:29.669528 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:35:29.669498    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4843da-2662-48ba-a82a-7c72e6e7816c" path="/var/lib/kubelet/pods/8c4843da-2662-48ba-a82a-7c72e6e7816c/volumes"
Jul 09 18:36:19.792826 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:19.792791    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"]
Jul 09 18:36:19.803020 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:36:19.802984    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:36:19.803020 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:36:19.803020 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:36:19.803020 ip-10-0-132-16 kubenswrapper[2576]:  > containerID="43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3" cmd=["/bin/sleep","15"]
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:19.803049    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" message="" err=<
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]:  >
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:36:19.803064    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" containerName="main" containerID="cri-o://43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3"
Jul 09 18:36:19.803376 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:19.803111    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" containerName="main" containerID="cri-o://43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3" gracePeriod=60
Jul 09 18:36:20.064286 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.064266    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:36:20.088422 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.088394    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-kserve-provision-location\") pod \"ce0621b6-098f-4596-94f6-76339db6efb5\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") "
Jul 09 18:36:20.088549 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.088499    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj462\" (UniqueName: \"kubernetes.io/projected/ce0621b6-098f-4596-94f6-76339db6efb5-kube-api-access-pj462\") pod \"ce0621b6-098f-4596-94f6-76339db6efb5\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") "
Jul 09 18:36:20.088612 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.088548    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce0621b6-098f-4596-94f6-76339db6efb5-tls-certs\") pod \"ce0621b6-098f-4596-94f6-76339db6efb5\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") "
Jul 09 18:36:20.088612 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.088598    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-tokenizer-uds\") pod \"ce0621b6-098f-4596-94f6-76339db6efb5\" (UID: \"ce0621b6-098f-4596-94f6-76339db6efb5\") "
Jul 09 18:36:20.089076 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.089024    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ce0621b6-098f-4596-94f6-76339db6efb5" (UID: "ce0621b6-098f-4596-94f6-76339db6efb5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:20.089202 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.089174    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce0621b6-098f-4596-94f6-76339db6efb5" (UID: "ce0621b6-098f-4596-94f6-76339db6efb5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:20.090551 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.090528    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0621b6-098f-4596-94f6-76339db6efb5-kube-api-access-pj462" (OuterVolumeSpecName: "kube-api-access-pj462") pod "ce0621b6-098f-4596-94f6-76339db6efb5" (UID: "ce0621b6-098f-4596-94f6-76339db6efb5"). InnerVolumeSpecName "kube-api-access-pj462". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:20.090667 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.090591    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0621b6-098f-4596-94f6-76339db6efb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce0621b6-098f-4596-94f6-76339db6efb5" (UID: "ce0621b6-098f-4596-94f6-76339db6efb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:20.189902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.189876    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj462\" (UniqueName: \"kubernetes.io/projected/ce0621b6-098f-4596-94f6-76339db6efb5-kube-api-access-pj462\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:36:20.189902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.189897    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce0621b6-098f-4596-94f6-76339db6efb5-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:36:20.189902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.189907    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-tokenizer-uds\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:36:20.190103 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.189917    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce0621b6-098f-4596-94f6-76339db6efb5-kserve-provision-location\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:36:20.684004 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.683971    2576 generic.go:356] "Generic (PLEG): container finished" podID="ce0621b6-098f-4596-94f6-76339db6efb5" containerID="43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3" exitCode=0
Jul 09 18:36:20.684218 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.684018    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"
Jul 09 18:36:20.684218 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.684068    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" event={"ID":"ce0621b6-098f-4596-94f6-76339db6efb5","Type":"ContainerDied","Data":"43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3"}
Jul 09 18:36:20.684218 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.684125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw" event={"ID":"ce0621b6-098f-4596-94f6-76339db6efb5","Type":"ContainerDied","Data":"53e8596af74635df2111d35a61317e3542036d594bc53f6c7915b8aba2d6f913"}
Jul 09 18:36:20.684218 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.684153    2576 scope.go:117] "RemoveContainer" containerID="43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3"
Jul 09 18:36:20.693539 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.693529    2576 scope.go:117] "RemoveContainer" containerID="c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08"
Jul 09 18:36:20.700995 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.700977    2576 scope.go:117] "RemoveContainer" containerID="43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3"
Jul 09 18:36:20.701277 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:36:20.701260    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3\": container with ID starting with 43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3 not found: ID does not exist" containerID="43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3"
Jul 09 18:36:20.701354 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.701284    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3"} err="failed to get container status \"43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3\": rpc error: code = NotFound desc = could not find container \"43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3\": container with ID starting with 43d8eee7a22c98c8692122bc18dfb282895e3ccda21b05bca7045279b943eec3 not found: ID does not exist"
Jul 09 18:36:20.701354 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.701300    2576 scope.go:117] "RemoveContainer" containerID="c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08"
Jul 09 18:36:20.701557 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:36:20.701540    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08\": container with ID starting with c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08 not found: ID does not exist" containerID="c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08"
Jul 09 18:36:20.701603 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.701565    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08"} err="failed to get container status \"c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08\": rpc error: code = NotFound desc = could not find container \"c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08\": container with ID starting with c823988d1f87490b9b006fc52fe14549e9b5dd8fe2eb48341871e23ad687da08 not found: ID does not exist"
Jul 09 18:36:20.705718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.705698    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"]
Jul 09 18:36:20.711518 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:20.711498    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5f5849dc8f-wlflw"]
Jul 09 18:36:21.673773 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:36:21.673742    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" path="/var/lib/kubelet/pods/ce0621b6-098f-4596-94f6-76339db6efb5/volumes"
Jul 09 18:37:01.410397 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410366    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"]
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410743    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4843da-2662-48ba-a82a-7c72e6e7816c" containerName="main"
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410754    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4843da-2662-48ba-a82a-7c72e6e7816c" containerName="main"
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410767    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" containerName="storage-initializer"
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410772    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" containerName="storage-initializer"
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410780    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" containerName="main"
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410785    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" containerName="main"
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410849    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce0621b6-098f-4596-94f6-76339db6efb5" containerName="main"
Jul 09 18:37:01.412891 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.410859    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4843da-2662-48ba-a82a-7c72e6e7816c" containerName="main"
Jul 09 18:37:01.413901 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.413886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.417564 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.417542    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 09 18:37:01.417682 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.417566    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-w95qr\""
Jul 09 18:37:01.425797 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.425772    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"]
Jul 09 18:37:01.521558 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.521518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/922e664b-f650-4a1d-bff0-e61e37bc4a87-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.521683 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.521582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922e664b-f650-4a1d-bff0-e61e37bc4a87-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.521683 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.521615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gjd\" (UniqueName: \"kubernetes.io/projected/922e664b-f650-4a1d-bff0-e61e37bc4a87-kube-api-access-42gjd\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.622464 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.622437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/922e664b-f650-4a1d-bff0-e61e37bc4a87-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.622594 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.622486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922e664b-f650-4a1d-bff0-e61e37bc4a87-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.622594 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.622524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gjd\" (UniqueName: \"kubernetes.io/projected/922e664b-f650-4a1d-bff0-e61e37bc4a87-kube-api-access-42gjd\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.622820 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.622803    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/922e664b-f650-4a1d-bff0-e61e37bc4a87-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.624973 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.624951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922e664b-f650-4a1d-bff0-e61e37bc4a87-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.632348 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.632328    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gjd\" (UniqueName: \"kubernetes.io/projected/922e664b-f650-4a1d-bff0-e61e37bc4a87-kube-api-access-42gjd\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.722931 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.722868    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:01.855389 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.855361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"]
Jul 09 18:37:01.857349 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:37:01.857323    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922e664b_f650_4a1d_bff0_e61e37bc4a87.slice/crio-d3a87029a73876c9ed1b61a7914a1f7959ac9216cbaf7bf36d187afec75ff8c3 WatchSource:0}: Error finding container d3a87029a73876c9ed1b61a7914a1f7959ac9216cbaf7bf36d187afec75ff8c3: Status 404 returned error can't find the container with id d3a87029a73876c9ed1b61a7914a1f7959ac9216cbaf7bf36d187afec75ff8c3
Jul 09 18:37:01.868311 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:01.868286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" event={"ID":"922e664b-f650-4a1d-bff0-e61e37bc4a87","Type":"ContainerStarted","Data":"d3a87029a73876c9ed1b61a7914a1f7959ac9216cbaf7bf36d187afec75ff8c3"}
Jul 09 18:37:02.873872 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:02.873839    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" event={"ID":"922e664b-f650-4a1d-bff0-e61e37bc4a87","Type":"ContainerStarted","Data":"e9c9a4b16c7112a4e0d71c147bf8466c486b51fad908c6d2f02ef28983c7ce7c"}
Jul 09 18:37:02.874273 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:02.873931    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:02.893961 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:02.893912    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" podStartSLOduration=1.8938988220000001 podStartE2EDuration="1.893898822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:02.892380964 +0000 UTC m=+3452.423158018" watchObservedRunningTime="2026-07-09 18:37:02.893898822 +0000 UTC m=+3452.424675852"
Jul 09 18:37:33.880351 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:33.880325    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:37:46.792377 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.792345    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"]
Jul 09 18:37:46.795738 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.795718    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.809008 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.808987    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"]
Jul 09 18:37:46.883729 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.883699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.883729 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.883733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.883937 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.883802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c71c41-1a32-47d6-b805-bc646bfdae59-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.883937 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.883851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.883937 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.883880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.884051 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.883940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc9dk\" (UniqueName: \"kubernetes.io/projected/96c71c41-1a32-47d6-b805-bc646bfdae59-kube-api-access-jc9dk\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.984609 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.984586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c71c41-1a32-47d6-b805-bc646bfdae59-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.984718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.984632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.984718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.984670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.984718 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.984710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc9dk\" (UniqueName: \"kubernetes.io/projected/96c71c41-1a32-47d6-b805-bc646bfdae59-kube-api-access-jc9dk\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.984859 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.984740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.984859 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.984766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.985172 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.985149    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.985172 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.985166    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.985286 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.985232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.986925 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.986897    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.987199 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.987169    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c71c41-1a32-47d6-b805-bc646bfdae59-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:46.994003 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:46.993983    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc9dk\" (UniqueName: \"kubernetes.io/projected/96c71c41-1a32-47d6-b805-bc646bfdae59-kube-api-access-jc9dk\") pod \"rolling-upgrade-test-kserve-859df57bcb-dpmdh\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:47.105557 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:47.105531    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:47.245725 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:47.245704    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"]
Jul 09 18:37:47.247490 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:37:47.247460    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c71c41_1a32_47d6_b805_bc646bfdae59.slice/crio-ba1e82e5ca8e77f326b3fd951bd9b272e4b4b88156f3d479c3b1c49ca07c33f9 WatchSource:0}: Error finding container ba1e82e5ca8e77f326b3fd951bd9b272e4b4b88156f3d479c3b1c49ca07c33f9: Status 404 returned error can't find the container with id ba1e82e5ca8e77f326b3fd951bd9b272e4b4b88156f3d479c3b1c49ca07c33f9
Jul 09 18:37:48.095115 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:48.095076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh" event={"ID":"96c71c41-1a32-47d6-b805-bc646bfdae59","Type":"ContainerStarted","Data":"0720d8b420b68a184103613ff7b44298d50228637c76225dfa2fd75589cc8070"}
Jul 09 18:37:48.095472 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:48.095157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh" event={"ID":"96c71c41-1a32-47d6-b805-bc646bfdae59","Type":"ContainerStarted","Data":"ba1e82e5ca8e77f326b3fd951bd9b272e4b4b88156f3d479c3b1c49ca07c33f9"}
Jul 09 18:37:48.106194 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:48.106173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:48.116587 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:48.116539    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh" podStartSLOduration=2.116526486 podStartE2EDuration="2.116526486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:48.115600836 +0000 UTC m=+3497.646377867" watchObservedRunningTime="2026-07-09 18:37:48.116526486 +0000 UTC m=+3497.647303515"
Jul 09 18:37:57.105916 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:57.105837    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:57.118170 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:57.118147    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:37:57.150640 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:37:57.150619    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:38:27.122270 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.122220    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"]
Jul 09 18:38:27.125879 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.125855    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"]
Jul 09 18:38:27.134006 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:38:27.133969    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:38:27.134006 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:38:27.134006 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:38:27.134006 ip-10-0-132-16 kubenswrapper[2576]:  > containerID="e9c9a4b16c7112a4e0d71c147bf8466c486b51fad908c6d2f02ef28983c7ce7c" cmd=["/bin/sleep","15"]
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.134029    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" message="" err=<
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]:  >
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]: E0709 18:38:27.134063    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" podUID="922e664b-f650-4a1d-bff0-e61e37bc4a87" containerName="main" containerID="cri-o://e9c9a4b16c7112a4e0d71c147bf8466c486b51fad908c6d2f02ef28983c7ce7c"
Jul 09 18:38:27.134249 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.134115    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" podUID="922e664b-f650-4a1d-bff0-e61e37bc4a87" containerName="main" containerID="cri-o://e9c9a4b16c7112a4e0d71c147bf8466c486b51fad908c6d2f02ef28983c7ce7c" gracePeriod=60
Jul 09 18:38:27.287406 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.287376    2576 generic.go:356] "Generic (PLEG): container finished" podID="922e664b-f650-4a1d-bff0-e61e37bc4a87" containerID="e9c9a4b16c7112a4e0d71c147bf8466c486b51fad908c6d2f02ef28983c7ce7c" exitCode=0
Jul 09 18:38:27.287567 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.287449    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" event={"ID":"922e664b-f650-4a1d-bff0-e61e37bc4a87","Type":"ContainerDied","Data":"e9c9a4b16c7112a4e0d71c147bf8466c486b51fad908c6d2f02ef28983c7ce7c"}
Jul 09 18:38:27.381677 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.381622    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:38:27.419846 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.419825    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42gjd\" (UniqueName: \"kubernetes.io/projected/922e664b-f650-4a1d-bff0-e61e37bc4a87-kube-api-access-42gjd\") pod \"922e664b-f650-4a1d-bff0-e61e37bc4a87\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") "
Jul 09 18:38:27.419968 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.419886    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922e664b-f650-4a1d-bff0-e61e37bc4a87-tls-certs\") pod \"922e664b-f650-4a1d-bff0-e61e37bc4a87\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") "
Jul 09 18:38:27.419968 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.419941    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/922e664b-f650-4a1d-bff0-e61e37bc4a87-tokenizer-uds\") pod \"922e664b-f650-4a1d-bff0-e61e37bc4a87\" (UID: \"922e664b-f650-4a1d-bff0-e61e37bc4a87\") "
Jul 09 18:38:27.420333 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.420303    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922e664b-f650-4a1d-bff0-e61e37bc4a87-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "922e664b-f650-4a1d-bff0-e61e37bc4a87" (UID: "922e664b-f650-4a1d-bff0-e61e37bc4a87"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:27.421778 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.421749    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922e664b-f650-4a1d-bff0-e61e37bc4a87-kube-api-access-42gjd" (OuterVolumeSpecName: "kube-api-access-42gjd") pod "922e664b-f650-4a1d-bff0-e61e37bc4a87" (UID: "922e664b-f650-4a1d-bff0-e61e37bc4a87"). InnerVolumeSpecName "kube-api-access-42gjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:38:27.421872 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.421824    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e664b-f650-4a1d-bff0-e61e37bc4a87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "922e664b-f650-4a1d-bff0-e61e37bc4a87" (UID: "922e664b-f650-4a1d-bff0-e61e37bc4a87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:38:27.520734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.520711    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922e664b-f650-4a1d-bff0-e61e37bc4a87-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:27.520734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.520732    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/922e664b-f650-4a1d-bff0-e61e37bc4a87-tokenizer-uds\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:27.520869 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:27.520741    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gjd\" (UniqueName: \"kubernetes.io/projected/922e664b-f650-4a1d-bff0-e61e37bc4a87-kube-api-access-42gjd\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:28.293115 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:28.293085    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"
Jul 09 18:38:28.293115 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:28.293111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h" event={"ID":"922e664b-f650-4a1d-bff0-e61e37bc4a87","Type":"ContainerDied","Data":"d3a87029a73876c9ed1b61a7914a1f7959ac9216cbaf7bf36d187afec75ff8c3"}
Jul 09 18:38:28.293566 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:28.293171    2576 scope.go:117] "RemoveContainer" containerID="e9c9a4b16c7112a4e0d71c147bf8466c486b51fad908c6d2f02ef28983c7ce7c"
Jul 09 18:38:28.309521 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:28.309496    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"]
Jul 09 18:38:28.312921 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:28.312900    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-s9q6h"]
Jul 09 18:38:29.669244 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:29.669210    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922e664b-f650-4a1d-bff0-e61e37bc4a87" path="/var/lib/kubelet/pods/922e664b-f650-4a1d-bff0-e61e37bc4a87/volumes"
Jul 09 18:38:42.147578 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.147515    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh" podUID="96c71c41-1a32-47d6-b805-bc646bfdae59" containerName="main" containerID="cri-o://0720d8b420b68a184103613ff7b44298d50228637c76225dfa2fd75589cc8070" gracePeriod=45
Jul 09 18:38:42.152584 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.152553    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh" podUID="96c71c41-1a32-47d6-b805-bc646bfdae59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": EOF"
Jul 09 18:38:42.360300 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.360270    2576 generic.go:356] "Generic (PLEG): container finished" podID="96c71c41-1a32-47d6-b805-bc646bfdae59" containerID="0720d8b420b68a184103613ff7b44298d50228637c76225dfa2fd75589cc8070" exitCode=0
Jul 09 18:38:42.360467 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.360352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh" event={"ID":"96c71c41-1a32-47d6-b805-bc646bfdae59","Type":"ContainerDied","Data":"0720d8b420b68a184103613ff7b44298d50228637c76225dfa2fd75589cc8070"}
Jul 09 18:38:42.406945 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.406891    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:38:42.543476 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543439    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c71c41-1a32-47d6-b805-bc646bfdae59-tls-certs\") pod \"96c71c41-1a32-47d6-b805-bc646bfdae59\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") "
Jul 09 18:38:42.543476 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543486    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-tmp-dir\") pod \"96c71c41-1a32-47d6-b805-bc646bfdae59\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") "
Jul 09 18:38:42.543675 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543503    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-model-cache\") pod \"96c71c41-1a32-47d6-b805-bc646bfdae59\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") "
Jul 09 18:38:42.543675 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543533    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-dshm\") pod \"96c71c41-1a32-47d6-b805-bc646bfdae59\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") "
Jul 09 18:38:42.543675 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543555    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-home\") pod \"96c71c41-1a32-47d6-b805-bc646bfdae59\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") "
Jul 09 18:38:42.543675 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543584    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc9dk\" (UniqueName: \"kubernetes.io/projected/96c71c41-1a32-47d6-b805-bc646bfdae59-kube-api-access-jc9dk\") pod \"96c71c41-1a32-47d6-b805-bc646bfdae59\" (UID: \"96c71c41-1a32-47d6-b805-bc646bfdae59\") "
Jul 09 18:38:42.543902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543763    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96c71c41-1a32-47d6-b805-bc646bfdae59" (UID: "96c71c41-1a32-47d6-b805-bc646bfdae59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:42.543902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543815    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-home" (OuterVolumeSpecName: "home") pod "96c71c41-1a32-47d6-b805-bc646bfdae59" (UID: "96c71c41-1a32-47d6-b805-bc646bfdae59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:42.543902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543852    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-model-cache" (OuterVolumeSpecName: "model-cache") pod "96c71c41-1a32-47d6-b805-bc646bfdae59" (UID: "96c71c41-1a32-47d6-b805-bc646bfdae59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:42.543902 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543892    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-home\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:42.544071 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.543907    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-tmp-dir\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:42.545600 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.545569    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-dshm" (OuterVolumeSpecName: "dshm") pod "96c71c41-1a32-47d6-b805-bc646bfdae59" (UID: "96c71c41-1a32-47d6-b805-bc646bfdae59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:42.545600 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.545585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c71c41-1a32-47d6-b805-bc646bfdae59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96c71c41-1a32-47d6-b805-bc646bfdae59" (UID: "96c71c41-1a32-47d6-b805-bc646bfdae59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:38:42.545734 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.545613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c71c41-1a32-47d6-b805-bc646bfdae59-kube-api-access-jc9dk" (OuterVolumeSpecName: "kube-api-access-jc9dk") pod "96c71c41-1a32-47d6-b805-bc646bfdae59" (UID: "96c71c41-1a32-47d6-b805-bc646bfdae59"). InnerVolumeSpecName "kube-api-access-jc9dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:38:42.644360 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.644330    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-dshm\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:42.644360 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.644355    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc9dk\" (UniqueName: \"kubernetes.io/projected/96c71c41-1a32-47d6-b805-bc646bfdae59-kube-api-access-jc9dk\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:42.644525 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.644366    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c71c41-1a32-47d6-b805-bc646bfdae59-tls-certs\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:42.644525 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:42.644375    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c71c41-1a32-47d6-b805-bc646bfdae59-model-cache\") on node \"ip-10-0-132-16.ec2.internal\" DevicePath \"\""
Jul 09 18:38:43.365862 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:43.365837    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"
Jul 09 18:38:43.365862 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:43.365845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh" event={"ID":"96c71c41-1a32-47d6-b805-bc646bfdae59","Type":"ContainerDied","Data":"ba1e82e5ca8e77f326b3fd951bd9b272e4b4b88156f3d479c3b1c49ca07c33f9"}
Jul 09 18:38:43.366339 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:43.365905    2576 scope.go:117] "RemoveContainer" containerID="0720d8b420b68a184103613ff7b44298d50228637c76225dfa2fd75589cc8070"
Jul 09 18:38:43.389064 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:43.389026    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"]
Jul 09 18:38:43.393695 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:43.393677    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-dpmdh"]
Jul 09 18:38:43.669633 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:43.669560    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c71c41-1a32-47d6-b805-bc646bfdae59" path="/var/lib/kubelet/pods/96c71c41-1a32-47d6-b805-bc646bfdae59/volumes"
Jul 09 18:38:47.826905 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.826869    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"]
Jul 09 18:38:47.827456 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.827437    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c71c41-1a32-47d6-b805-bc646bfdae59" containerName="main"
Jul 09 18:38:47.827506 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.827460    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c71c41-1a32-47d6-b805-bc646bfdae59" containerName="main"
Jul 09 18:38:47.827548 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.827509    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e664b-f650-4a1d-bff0-e61e37bc4a87" containerName="main"
Jul 09 18:38:47.827548 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.827519    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e664b-f650-4a1d-bff0-e61e37bc4a87" containerName="main"
Jul 09 18:38:47.827621 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.827601    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c71c41-1a32-47d6-b805-bc646bfdae59" containerName="main"
Jul 09 18:38:47.827621 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.827617    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="922e664b-f650-4a1d-bff0-e61e37bc4a87" containerName="main"
Jul 09 18:38:47.838080 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.838009    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:47.841227 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.841203    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"]
Jul 09 18:38:47.841836 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.841816    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:38:47.842207 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.842185    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 09 18:38:47.842207 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.842190    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:38:47.842377 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.842204    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kf2gt\""
Jul 09 18:38:47.985431 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.985402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49134e30-2b40-421c-91bb-5fecce9292e0-cert\") pod \"llmisvc-controller-manager-7cd64959bc-vgm6v\" (UID: \"49134e30-2b40-421c-91bb-5fecce9292e0\") " pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:47.985544 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:47.985446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzl4\" (UniqueName: \"kubernetes.io/projected/49134e30-2b40-421c-91bb-5fecce9292e0-kube-api-access-rjzl4\") pod \"llmisvc-controller-manager-7cd64959bc-vgm6v\" (UID: \"49134e30-2b40-421c-91bb-5fecce9292e0\") " pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:48.086341 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.086282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49134e30-2b40-421c-91bb-5fecce9292e0-cert\") pod \"llmisvc-controller-manager-7cd64959bc-vgm6v\" (UID: \"49134e30-2b40-421c-91bb-5fecce9292e0\") " pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:48.086341 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.086323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzl4\" (UniqueName: \"kubernetes.io/projected/49134e30-2b40-421c-91bb-5fecce9292e0-kube-api-access-rjzl4\") pod \"llmisvc-controller-manager-7cd64959bc-vgm6v\" (UID: \"49134e30-2b40-421c-91bb-5fecce9292e0\") " pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:48.088666 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.088636    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49134e30-2b40-421c-91bb-5fecce9292e0-cert\") pod \"llmisvc-controller-manager-7cd64959bc-vgm6v\" (UID: \"49134e30-2b40-421c-91bb-5fecce9292e0\") " pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:48.097881 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.097859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzl4\" (UniqueName: \"kubernetes.io/projected/49134e30-2b40-421c-91bb-5fecce9292e0-kube-api-access-rjzl4\") pod \"llmisvc-controller-manager-7cd64959bc-vgm6v\" (UID: \"49134e30-2b40-421c-91bb-5fecce9292e0\") " pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:48.149413 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.149392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:48.281880 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.281852    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"]
Jul 09 18:38:48.285319 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:38:48.285289    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49134e30_2b40_421c_91bb_5fecce9292e0.slice/crio-a4a1f87c4a5857fb74ee06ec02651d9246e748a51da105b9338dadd7f300cd86 WatchSource:0}: Error finding container a4a1f87c4a5857fb74ee06ec02651d9246e748a51da105b9338dadd7f300cd86: Status 404 returned error can't find the container with id a4a1f87c4a5857fb74ee06ec02651d9246e748a51da105b9338dadd7f300cd86
Jul 09 18:38:48.286725 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.286708    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:38:48.391291 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:48.391235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v" event={"ID":"49134e30-2b40-421c-91bb-5fecce9292e0","Type":"ContainerStarted","Data":"a4a1f87c4a5857fb74ee06ec02651d9246e748a51da105b9338dadd7f300cd86"}
Jul 09 18:38:52.413528 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:52.413496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v" event={"ID":"49134e30-2b40-421c-91bb-5fecce9292e0","Type":"ContainerStarted","Data":"243c2f40efec41cd65ee9b3ec16399a804b74110677327bd004697007c110935"}
Jul 09 18:38:52.413909 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:52.413566    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:38:52.432886 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:38:52.432840    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v" podStartSLOduration=1.715890902 podStartE2EDuration="5.432825406s" totalImagesPullingTime="3.716934504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:38:52.431737606 +0000 UTC m=+3561.962514661" watchObservedRunningTime="2026-07-09 18:38:52.432825406 +0000 UTC m=+3561.963602468"
Jul 09 18:39:23.420949 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:39:23.420881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7cd64959bc-vgm6v"
Jul 09 18:39:31.284641 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:39:31.284612    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:39:31.294149 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:39:31.294129    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jwnj6_dd4e8426-56fe-4848-9f7a-7d00b5108aa7/ovn-acl-logging/0.log"
Jul 09 18:40:18.248018 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:18.247992    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:18.253679 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:18.253655    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:19.940056 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:19.940023    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:19.947168 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:19.947147    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:21.607740 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:21.607700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:21.615251 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:21.615228    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:23.219729 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:23.219691    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:23.226280 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:23.226259    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:24.952312 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:24.952281    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:24.958547 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:24.958526    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:26.560737 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:26.560707    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:26.576635 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:26.576591    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:28.265975 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:28.265947    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:28.284642 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:28.284618    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:30.007454 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:30.007419    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:30.019079 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:30.019057    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:31.676357 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:31.676323    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:31.684184 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:31.684165    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:33.353392 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:33.353364    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:33.359866 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:33.359847    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:34.978248 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:34.978210    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:34.986695 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:34.986676    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:36.648923 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:36.648894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:36.655642 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:36.655621    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:38.272882 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:38.272852    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:38.281794 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:38.281768    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:39.812699 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:39.812665    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:39.826502 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:39.826478    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:41.457349 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:41.457318    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:41.463862 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:41.463823    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:43.062783 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:43.062752    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/main/0.log"
Jul 09 18:40:43.069118 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:43.069097    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv_ce9d7d7f-5c2b-4845-a794-a578803d5d9a/storage-initializer/0.log"
Jul 09 18:40:52.533994 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:52.533927    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:40:52.553005 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:52.552978    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:40:53.966637 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:53.966612    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:40:53.985395 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:53.985372    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:40:55.415219 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:55.415191    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:40:55.430274 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:55.430253    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:40:56.791932 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:56.791900    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:40:56.809104 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:56.809080    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:40:58.199720 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:58.199688    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:40:58.215384 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:58.215362    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:40:59.574355 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:59.574331    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:40:59.592011 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:40:59.591988    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:41:00.978201 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:00.978174    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:41:00.992370 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:00.992349    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:41:02.391897 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:02.391873    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:41:02.408247 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:02.408221    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:41:03.819598 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:03.819571    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:41:03.835492 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:03.835473    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:41:05.229442 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:05.229416    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:41:05.246919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:05.246894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:41:06.621225 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:06.621200    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:41:06.638346 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:06.638326    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:41:12.354919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:12.354889    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rwpm5_01ffe74e-2c33-429c-8818-fb8d7fa9f969/global-pull-secret-syncer/0.log"
Jul 09 18:41:12.468012 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:12.467986    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d6l4b_a87002de-8bc9-4f3c-8dad-fb393e549cf1/konnectivity-agent/0.log"
Jul 09 18:41:12.528421 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:12.528400    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-16.ec2.internal_c2560b51e81baa57e27e9a93c886cb16/haproxy/0.log"
Jul 09 18:41:18.713439 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:18.713413    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-t6fxz_5cc22cd2-6536-4fdb-b3bf-adc56315c8a2/limitador/0.log"
Jul 09 18:41:18.768585 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:18.768557    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-xh7g7_3aefcf51-79c0-4275-aa20-638cbabd5900/manager/0.log"
Jul 09 18:41:20.213288 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.213257    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c683809c-af44-4756-9c3c-63e979924bda/alertmanager/0.log"
Jul 09 18:41:20.245977 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.245953    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c683809c-af44-4756-9c3c-63e979924bda/config-reloader/0.log"
Jul 09 18:41:20.293920 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.293837    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c683809c-af44-4756-9c3c-63e979924bda/kube-rbac-proxy-web/0.log"
Jul 09 18:41:20.323049 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.323013    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c683809c-af44-4756-9c3c-63e979924bda/kube-rbac-proxy/0.log"
Jul 09 18:41:20.355949 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.355917    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c683809c-af44-4756-9c3c-63e979924bda/kube-rbac-proxy-metric/0.log"
Jul 09 18:41:20.387759 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.387734    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c683809c-af44-4756-9c3c-63e979924bda/prom-label-proxy/0.log"
Jul 09 18:41:20.418407 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.418382    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c683809c-af44-4756-9c3c-63e979924bda/init-config-reloader/0.log"
Jul 09 18:41:20.517012 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.516953    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-98d5p_c01b5c16-251b-4550-baeb-11db543cd02a/kube-state-metrics/0.log"
Jul 09 18:41:20.552504 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.552436    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-98d5p_c01b5c16-251b-4550-baeb-11db543cd02a/kube-rbac-proxy-main/0.log"
Jul 09 18:41:20.580444 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.580396    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-98d5p_c01b5c16-251b-4550-baeb-11db543cd02a/kube-rbac-proxy-self/0.log"
Jul 09 18:41:20.615900 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.615879    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5975c47848-c26vd_b0ac9b34-3e1e-40e2-bda2-0d3a0ea6c2c8/metrics-server/0.log"
Jul 09 18:41:20.647914 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.647890    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-x7rcz_d5fea503-e58d-4f3c-8e54-9e5441ab0202/monitoring-plugin/0.log"
Jul 09 18:41:20.873280 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.873250    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x6v6k_2b46e220-837a-4114-969e-84fa58778b51/node-exporter/0.log"
Jul 09 18:41:20.911013 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.910982    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x6v6k_2b46e220-837a-4114-969e-84fa58778b51/kube-rbac-proxy/0.log"
Jul 09 18:41:20.954851 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:20.954831    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x6v6k_2b46e220-837a-4114-969e-84fa58778b51/init-textfile/0.log"
Jul 09 18:41:21.014943 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.014923    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-v6nj5_24b99d10-5d35-406e-8805-4a9a06009a5d/kube-rbac-proxy-main/0.log"
Jul 09 18:41:21.037207 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.037187    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-v6nj5_24b99d10-5d35-406e-8805-4a9a06009a5d/kube-rbac-proxy-self/0.log"
Jul 09 18:41:21.069963 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.069947    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-v6nj5_24b99d10-5d35-406e-8805-4a9a06009a5d/openshift-state-metrics/0.log"
Jul 09 18:41:21.108032 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.108010    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32ea468c-e785-4180-bdb1-a473f520f40c/prometheus/0.log"
Jul 09 18:41:21.127930 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.127883    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32ea468c-e785-4180-bdb1-a473f520f40c/config-reloader/0.log"
Jul 09 18:41:21.155415 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.155395    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32ea468c-e785-4180-bdb1-a473f520f40c/thanos-sidecar/0.log"
Jul 09 18:41:21.184579 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.184563    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32ea468c-e785-4180-bdb1-a473f520f40c/kube-rbac-proxy-web/0.log"
Jul 09 18:41:21.217140 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.217125    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32ea468c-e785-4180-bdb1-a473f520f40c/kube-rbac-proxy/0.log"
Jul 09 18:41:21.247951 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.247935    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32ea468c-e785-4180-bdb1-a473f520f40c/kube-rbac-proxy-thanos/0.log"
Jul 09 18:41:21.278737 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.278718    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32ea468c-e785-4180-bdb1-a473f520f40c/init-config-reloader/0.log"
Jul 09 18:41:21.321825 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.321806    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xqhrm_d53c755e-ee57-4491-ad8b-11081e828c6c/prometheus-operator/0.log"
Jul 09 18:41:21.359810 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.359790    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xqhrm_d53c755e-ee57-4491-ad8b-11081e828c6c/kube-rbac-proxy/0.log"
Jul 09 18:41:21.397116 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.397068    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-khplm_03d952ac-152f-427f-85cd-9672adcc63db/prometheus-operator-admission-webhook/0.log"
Jul 09 18:41:21.438434 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.438416    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5449c784b4-nglgv_c4d402be-dc3c-482c-83b4-bcfd421ca489/telemeter-client/0.log"
Jul 09 18:41:21.465192 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.465170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5449c784b4-nglgv_c4d402be-dc3c-482c-83b4-bcfd421ca489/reload/0.log"
Jul 09 18:41:21.493950 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.493934    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5449c784b4-nglgv_c4d402be-dc3c-482c-83b4-bcfd421ca489/kube-rbac-proxy/0.log"
Jul 09 18:41:21.534544 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.534525    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6779965f77-zwksh_8f5e947c-e2d1-4028-8fb5-b695d7716e0e/thanos-query/0.log"
Jul 09 18:41:21.567132 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.567115    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6779965f77-zwksh_8f5e947c-e2d1-4028-8fb5-b695d7716e0e/kube-rbac-proxy-web/0.log"
Jul 09 18:41:21.592749 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.592730    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6779965f77-zwksh_8f5e947c-e2d1-4028-8fb5-b695d7716e0e/kube-rbac-proxy/0.log"
Jul 09 18:41:21.615942 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.615922    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6779965f77-zwksh_8f5e947c-e2d1-4028-8fb5-b695d7716e0e/prom-label-proxy/0.log"
Jul 09 18:41:21.640741 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.640717    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6779965f77-zwksh_8f5e947c-e2d1-4028-8fb5-b695d7716e0e/kube-rbac-proxy-rules/0.log"
Jul 09 18:41:21.665026 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:21.664974    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6779965f77-zwksh_8f5e947c-e2d1-4028-8fb5-b695d7716e0e/kube-rbac-proxy-metrics/0.log"
Jul 09 18:41:23.377589 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.377552    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"]
Jul 09 18:41:23.381614 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.381598    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.385096 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.385077    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlxbw\"/\"openshift-service-ca.crt\""
Jul 09 18:41:23.385649 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.385625    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlxbw\"/\"kube-root-ca.crt\""
Jul 09 18:41:23.385762 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.385746    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xlxbw\"/\"default-dockercfg-wbfr7\""
Jul 09 18:41:23.390782 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.390759    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"]
Jul 09 18:41:23.431643 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.431621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-lib-modules\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.431740 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.431648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-proc\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.431789 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.431739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grppp\" (UniqueName: \"kubernetes.io/projected/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-kube-api-access-grppp\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.431789 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.431757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-sys\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.431789 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.431773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-podres\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.532924 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.532894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grppp\" (UniqueName: \"kubernetes.io/projected/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-kube-api-access-grppp\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533113 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.532934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-sys\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533113 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.532963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-podres\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533113 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.532993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-lib-modules\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533113 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.533019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-proc\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533113 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.533032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-sys\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533113 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.533112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-proc\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533362 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.533160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-podres\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.533362 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.533187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-lib-modules\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.540919 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.540895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grppp\" (UniqueName: \"kubernetes.io/projected/faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3-kube-api-access-grppp\") pod \"perf-node-gather-daemonset-hsl6w\" (UID: \"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.692028 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.691957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:23.824946 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:23.824921    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"]
Jul 09 18:41:23.827485 ip-10-0-132-16 kubenswrapper[2576]: W0709 18:41:23.827455    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfaf1c6a3_4fb7_4e9b_b8eb_fa750d7b6eb3.slice/crio-cb6ebc27901d4312c17699e44c83d996bf51589bbcb0ec91f68c129a7f946d9a WatchSource:0}: Error finding container cb6ebc27901d4312c17699e44c83d996bf51589bbcb0ec91f68c129a7f946d9a: Status 404 returned error can't find the container with id cb6ebc27901d4312c17699e44c83d996bf51589bbcb0ec91f68c129a7f946d9a
Jul 09 18:41:24.124281 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:24.124251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w" event={"ID":"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3","Type":"ContainerStarted","Data":"2ed0c35b06fc18fb6033e662ba8f26ca176ece79c6687b5c742dff58208633c7"}
Jul 09 18:41:24.124409 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:24.124298    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w" event={"ID":"faf1c6a3-4fb7-4e9b-b8eb-fa750d7b6eb3","Type":"ContainerStarted","Data":"cb6ebc27901d4312c17699e44c83d996bf51589bbcb0ec91f68c129a7f946d9a"}
Jul 09 18:41:24.124409 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:24.124326    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:24.151518 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:24.151473    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w" podStartSLOduration=1.15145819 podStartE2EDuration="1.15145819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:41:24.148302898 +0000 UTC m=+3713.679079940" watchObservedRunningTime="2026-07-09 18:41:24.15145819 +0000 UTC m=+3713.682235220"
Jul 09 18:41:25.973440 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:25.973414    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dzsz9_1aeeb33f-1695-44b2-a531-6430ee5619de/dns/0.log"
Jul 09 18:41:25.996504 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:25.996481    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dzsz9_1aeeb33f-1695-44b2-a531-6430ee5619de/kube-rbac-proxy/0.log"
Jul 09 18:41:26.052209 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:26.052190    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dd2pj_6107f0d9-9d5a-4e33-b248-eb8b3ad63fe4/dns-node-resolver/0.log"
Jul 09 18:41:26.632404 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:26.632377    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7dbc88fcb8-4rjtv_82621485-6679-4d1a-9732-09d4db44ce8b/registry/0.log"
Jul 09 18:41:26.660141 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:26.660114    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5qmgm_928a4afe-36fc-4397-9e2b-d88e2156fff4/node-ca/0.log"
Jul 09 18:41:29.333170 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:29.333144    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-znkdl_130ee9fd-9318-4595-843a-484b2747d946/serve-healthcheck-canary/0.log"
Jul 09 18:41:29.939173 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:29.939142    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fsgn4_bf5c6f12-38fd-4951-bf72-18aecc136c1f/kube-rbac-proxy/0.log"
Jul 09 18:41:29.963119 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:29.963100    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fsgn4_bf5c6f12-38fd-4951-bf72-18aecc136c1f/exporter/0.log"
Jul 09 18:41:29.986202 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:29.986184    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fsgn4_bf5c6f12-38fd-4951-bf72-18aecc136c1f/extractor/0.log"
Jul 09 18:41:30.138757 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:30.138735    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-hsl6w"
Jul 09 18:41:33.299561 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:33.299482    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cd758bd69-ld6ng_97a21e99-4e62-4b47-b1fb-c330a64780a6/manager/0.log"
Jul 09 18:41:34.230889 ip-10-0-132-16 kubenswrapper[2576]: I0709 18:41:34.230820    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-7cd64959bc-vgm6v_49134e30-2b40-421c-91bb-5fecce9292e0/manager/0.log"