Jul 06 20:32:51.394815 ip-10-0-131-57 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 20:32:51.394825 ip-10-0-131-57 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 20:32:51.394833 ip-10-0-131-57 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 20:32:51.395205 ip-10-0-131-57 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 20:33:01.418884 ip-10-0-131-57 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 20:33:01.418903 ip-10-0-131-57 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2ac8f8f40a7a4417a041ded2d65acafc --
Jul 06 20:35:00.978878 ip-10-0-131-57 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.402520    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409667    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409682    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409687    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409690    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409694    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409699    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:01.477940 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409703    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409706    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409709    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409712    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409715    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409718    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409722    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409732    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409735    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409738    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409741    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409744    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409746    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409750    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409753    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409756    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409759    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409762    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409765    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409768    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:01.479445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409770    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409773    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409776    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409779    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409782    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409784    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409786    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409789    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409791    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409796    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409800    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409802    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409805    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409808    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409810    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409813    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409815    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409817    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409821    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409823    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:01.480065 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409825    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409830    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409833    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409836    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409838    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409842    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409845    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409847    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409850    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409853    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409855    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409858    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409860    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409863    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409866    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409869    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409872    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409874    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409877    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:01.480684 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409879    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409882    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409884    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409887    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409889    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409892    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409895    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409900    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409902    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409905    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409907    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409910    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409913    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409916    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409919    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409921    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409924    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409927    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409929    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409932    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:01.483876 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409934    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409938    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409940    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409943    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409945    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409948    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409951    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409953    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409956    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409959    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409961    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409964    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409966    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.409969    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411568    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411596    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411600    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411603    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411605    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411608    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:01.484917 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411611    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411614    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411617    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411619    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411622    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411625    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411628    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411631    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411633    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411636    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411639    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411642    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411645    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411647    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411650    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411653    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411657    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411660    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411664    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411666    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:01.488863 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411669    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411672    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411675    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411678    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411682    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411685    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411687    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411690    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411693    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411697    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411701    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411704    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411706    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411709    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411712    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411715    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411718    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411721    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411724    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:01.489668 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411727    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411730    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411732    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411735    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411738    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411740    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411743    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411745    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411748    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411751    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411753    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411755    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411758    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411760    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411763    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411765    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411768    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411770    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411772    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411775    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:01.490424 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411779    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:01.490833 ip-10-0-131-57 systemd[1]: Started Kubernetes Kubelet.
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411781    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411785    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411787    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411791    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411794    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411797    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411799    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411802    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411804    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411807    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411810    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411814    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411817    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411819    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411822    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411824    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411827    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411829    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411832    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:01.491222 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411835    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411837    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411839    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411842    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411844    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411847    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411850    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411852    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411855    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411857    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411860    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411865    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411868    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.411870    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412253    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412257    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412260    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412263    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412266    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412268    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:03.118686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412271    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412274    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412277    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412279    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412282    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412285    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412288    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412291    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412294    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412297    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412299    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412302    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412304    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412307    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412309    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412312    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412314    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412319    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412322    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:03.119680 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412325    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412328    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412331    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412333    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412338    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412341    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412343    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412346    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412348    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412351    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412353    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412355    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412358    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412361    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412364    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412367    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412371    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412375    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412378    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412381    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:03.120604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412384    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412387    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412390    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412393    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412395    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412398    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412400    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412403    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412406    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412409    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412412    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412414    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412417    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412419    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412422    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412425    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412428    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412431    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412433    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412436    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:03.121340 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412439    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412441    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412444    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412446    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412450    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412452    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412455    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412457    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412459    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412462    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412464    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412467    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412470    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412473    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412475    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412478    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412480    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412482    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412485    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412487    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:03.121921 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412490    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412492    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412495    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412499    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412501    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412504    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412507    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412509    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412513    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412518    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412521    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412524    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412526    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.412529    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412613    2582 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412621    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412627    2582 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412632    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412636    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412640    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412644    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 20:35:03.122821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412648    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412651    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412654    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412658    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412661    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412664    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412667    2582 flags.go:64] FLAG: --cgroup-root=""
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412670    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412673    2582 flags.go:64] FLAG: --client-ca-file=""
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412675    2582 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412678    2582 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412683    2582 flags.go:64] FLAG: --cluster-domain=""
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412685    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412688    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412691    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412695    2582 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412698    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412701    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412705    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412709    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412714    2582 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412717    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412720    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412723    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 20:35:03.123458 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412726    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412730    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412733    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412736    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412739    2582 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412742    2582 flags.go:64] FLAG: --enable-server="true"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412745    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412750    2582 flags.go:64] FLAG: --event-burst="100"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412753    2582 flags.go:64] FLAG: --event-qps="50"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412755    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412758    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412761    2582 flags.go:64] FLAG: --eviction-hard=""
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412766    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412769    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412771    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412774    2582 flags.go:64] FLAG: --eviction-soft=""
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412777    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412780    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412783    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412785    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412788    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412791    2582 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412794    2582 flags.go:64] FLAG: --feature-gates=""
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412798    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412800    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 20:35:03.124298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412804    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412806    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412811    2582 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412815    2582 flags.go:64] FLAG: --help="false"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412818    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412821    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412824    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412827    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412830    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412834    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412836    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412839    2582 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412842    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412845    2582 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412848    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412851    2582 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412854    2582 flags.go:64] FLAG: --kube-reserved=""
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412857    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412859    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412862    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412866    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412868    2582 flags.go:64] FLAG: --lock-file=""
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412871    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412874    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 20:35:03.125129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412877    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412883    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412886    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412888    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412891    2582 flags.go:64] FLAG: --logging-format="text"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412894    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412897    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412900    2582 flags.go:64] FLAG: --manifest-url=""
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412902    2582 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412907    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412911    2582 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412921    2582 flags.go:64] FLAG: --max-pods="110"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412924    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412926    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412929    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412932    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412935    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412938    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412941    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412956    2582 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412959    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412962    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412965    2582 flags.go:64] FLAG: --pod-cidr=""
Jul 06 20:35:03.125941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412968    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412973    2582 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412976    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412979    2582 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412982    2582 flags.go:64] FLAG: --port="10250"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412985    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412989    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060658ce6ea01962e"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412992    2582 flags.go:64] FLAG: --qos-reserved=""
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412995    2582 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.412998    2582 flags.go:64] FLAG: --register-node="true"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413001    2582 flags.go:64] FLAG: --register-with-taints=""
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413004    2582 flags.go:64] FLAG: --registry-burst="10"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413007    2582 flags.go:64] FLAG: --registry-qps="5"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413010    2582 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413013    2582 flags.go:64] FLAG: --reserved-memory=""
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413016    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413019    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413022    2582 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413025    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413029    2582 flags.go:64] FLAG: --runonce="false"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413032    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413037    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413039    2582 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413042    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413045    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413048    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 20:35:03.127002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413051    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413054    2582 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413057    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413060    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413063    2582 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413065    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413068    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413072    2582 flags.go:64] FLAG: --system-cgroups=""
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413074    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413079    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413082    2582 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413085    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413089    2582 flags.go:64] FLAG: --tls-min-version=""
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413093    2582 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413095    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413098    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413101    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413103    2582 flags.go:64] FLAG: --v="2"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413107    2582 flags.go:64] FLAG: --version="false"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413111    2582 flags.go:64] FLAG: --vmodule=""
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413115    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.413118    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413234    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413238    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413241    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:03.127804 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413246    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413249    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413253    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413256    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413259    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413261    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413264    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413267    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413270    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413272    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413275    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413277    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413280    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413282    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413285    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413288    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413290    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413294    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413297    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413300    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:03.128503 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413303    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413306    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413308    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413311    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413313    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413315    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413318    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413321    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413323    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413325    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413328    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413330    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413335    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413339    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413343    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413346    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413350    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413352    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413355    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:03.129100 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413358    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413361    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413364    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413367    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413369    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413372    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413375    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413378    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413380    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413383    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413385    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413388    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413390    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413393    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413395    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413400    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413402    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413405    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413407    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:03.129610 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413409    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413412    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413414    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413417    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413419    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413422    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413425    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413428    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413432    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413435    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413437    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413441    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413444    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413447    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413449    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413452    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413454    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413457    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413460    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:03.130167 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413463    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413465    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413468    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413471    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413474    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413477    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413479    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413482    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413484    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413487    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413490    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413492    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413495    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413497    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413500    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413502    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413504    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413507    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:03.130946 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.413509    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.414049    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.422531    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.422548    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.423416    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.423424    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423483    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423488    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423491    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423494    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423496    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423499    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423502    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423505    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423507    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:03.131464 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423510    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423513    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423516    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423518    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423522    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423528    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423531    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423534    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423537    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423540    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423542    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423547    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423549    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423553    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423555    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423558    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423560    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423563    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423566    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423568    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:03.131977 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423571    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423587    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423592    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423595    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423597    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423600    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423603    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423606    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423608    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423611    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423614    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423616    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423619    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423621    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423624    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423627    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423629    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423633    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423635    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423638    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:03.132509 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423641    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423643    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423646    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423650    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423653    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423655    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423658    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423660    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423663    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423666    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423668    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423671    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423673    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423676    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423678    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423681    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423683    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423686    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423688    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423691    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:03.133118 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423694    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423696    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423699    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423701    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423705    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423709    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423712    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423715    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423717    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423721    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423725    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423728    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423731    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423734    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423736    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423740    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423742    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423745    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423748    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:03.133631 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423750    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423753    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423755    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423758    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423762    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423766    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423769    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423772    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423775    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423777    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423780    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.423785    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423898    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423903    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:03.268042 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423906    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423909    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423912    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423915    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423918    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423921    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423924    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423926    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423931    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423935    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423938    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423940    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423943    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423946    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423948    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423952    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423954    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423957    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423960    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:03.285372 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423962    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423965    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423968    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423970    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423973    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423975    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423978    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423980    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423983    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423986    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423988    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423991    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423994    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423997    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.423999    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424002    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424006    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424009    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424012    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424014    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:03.286164 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424017    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424022    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424025    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424027    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424031    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424034    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424037    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424040    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424043    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424046    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424049    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424051    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424054    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424056    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424059    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424061    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424066    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424069    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424072    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424075    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:03.286739 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424078    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424081    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424083    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424086    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424089    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424092    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424094    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424097    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424099    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424102    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424104    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424107    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424109    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424113    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424115    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424118    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424120    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424124    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424126    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424129    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:03.287415 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424133    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424135    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424138    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424140    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424143    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424145    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424148    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424151    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424153    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424155    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424158    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424160    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424163    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424165    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424168    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424171    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424173    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:03.288445 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:01.424175    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.424180    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.424294    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.426434    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.427685    2582 server.go:1019] "Starting client certificate rotation"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.427785    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.427812    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.431506    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.436410    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.443780    2582 log.go:25] "Validated CRI v1 runtime API"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.448251    2582 log.go:25] "Validated CRI v1 image API"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.449110    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 20:35:03.289005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.453222    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c0db8a3e-936d-4a00-8937-48466aace162:/dev/nvme0n1p4 d5bbad84-da2d-4a74-9d03-e483c5975593:/dev/nvme0n1p3]
Jul 06 20:35:03.289398 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.453237    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.460453    2582 manager.go:216] Machine: {Timestamp:2026-07-06 20:35:01.458719125 +0000 UTC m=+0.358445229 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098540 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d2fb4a99c86ca20e1afaeb01d6eb4 SystemUUID:ec2d2fb4-a99c-86ca-20e1-afaeb01d6eb4 BootID:2ac8f8f4-0a7a-4417-a041-ded2d65acafc Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:58:4a:16:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:58:4a:16:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:2e:af:fb:95:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.460549    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.460643    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.463645    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.463668    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.464587    2582 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.464598    2582 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 20:35:03.289571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.464605    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.465334    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.466641    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.469549    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.469674    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.470394    2582 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.470406    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.470421    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.470431    2582 kubelet.go:398] "Adding apiserver pod source"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.470439    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.470820    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.470836    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.474427    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.475093    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.476419    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.476446    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483739    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483754    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483760    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483765    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483775    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483782    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483790    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483796    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483802    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483811    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483820    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.483825    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.484826    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.484834    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:01.485567    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:01.485616    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.488072    2582 server.go:1268] "Started kubelet"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.488165    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.488242    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.488333    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.490095    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.492227    2582 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:01.494208    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 20:35:03.289887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.503244    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:01.503177    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18bfcda89a3f585d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-07-06 20:35:01.488048221 +0000 UTC m=+0.387774324,LastTimestamp:2026-07-06 20:35:01.488048221 +0000 UTC m=+0.387774324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.504551    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.504609    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:01.504642    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505341    2582 factory.go:55] Registering systemd factory
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505356    2582 factory.go:223] Registration of the systemd container factory successfully
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505372    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505383    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505369    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505545    2582 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505556    2582 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505567    2582 factory.go:153] Registering CRI-O factory
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505593    2582 factory.go:223] Registration of the crio container factory successfully
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505654    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505677    2582 factory.go:103] Registering Raw factory
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.505689    2582 manager.go:1196] Started watching for new ooms in manager
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.506146    2582 manager.go:349] Starting recovery of all containers
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:01.507567    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6f7j"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.080505    2582 manager.go:354] Recovery completed
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.085023    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.090167    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6f7j"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.090922    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.090951    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.090979    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.091637    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.091650    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.091666    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.091670    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.092408    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.092431    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.092444    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.093215    2582 policy_none.go:49] "None policy: Start"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.093228    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.093238    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.094230    2582 policy_none.go:47] "Start"
Jul 06 20:35:03.290974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.100388    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.116390    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-57.ec2.internal\" not found" node="ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.136573    2582 manager.go:337] "Starting Device Plugin manager"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.136606    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.136614    2582 server.go:87] "Starting device plugin registration server"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.136841    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.136850    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.136981    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.137076    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.137085    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.137505    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.137545    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.237163    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.238266    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.238295    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.238305    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.238331    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.254346    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.254367    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-57.ec2.internal\": node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.292123 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.287327    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.328749    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.330064    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.330099    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.330122    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.330156    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.335254    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.387809    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.430999    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"]
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.431082    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.432061    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.432086    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.432097    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.433434    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.433558    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.433584    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.434399    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.434425    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.434441    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.434975    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.435003    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.435017    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.435644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.435669    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.436351    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.436375    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:03.440678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.436385    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:03.556158 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.475641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb97d123bad45427ccd732f341cd5be2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"bb97d123bad45427ccd732f341cd5be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.556158 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.479696    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-57.ec2.internal\" not found" node="ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.556158 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.484411    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-57.ec2.internal\" not found" node="ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.556158 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.488283    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.576652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb97d123bad45427ccd732f341cd5be2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"bb97d123bad45427ccd732f341cd5be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.576682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb97d123bad45427ccd732f341cd5be2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"bb97d123bad45427ccd732f341cd5be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.576700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f22a2c57117da9e2036a061b6a5bcbad-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.ec2.internal\" (UID: \"f22a2c57117da9e2036a061b6a5bcbad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.576765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb97d123bad45427ccd732f341cd5be2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"bb97d123bad45427ccd732f341cd5be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.588911    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.676930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb97d123bad45427ccd732f341cd5be2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"bb97d123bad45427ccd732f341cd5be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.676974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f22a2c57117da9e2036a061b6a5bcbad-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.ec2.internal\" (UID: \"f22a2c57117da9e2036a061b6a5bcbad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.677009    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f22a2c57117da9e2036a061b6a5bcbad-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.ec2.internal\" (UID: \"f22a2c57117da9e2036a061b6a5bcbad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.677030    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb97d123bad45427ccd732f341cd5be2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"bb97d123bad45427ccd732f341cd5be2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.699927 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.688996    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.857324 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.784505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.857324 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.789096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:03.857324 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.789119    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.991321 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.889549    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.991321 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:03.990062    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:03.992512 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.992135    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 20:35:03.992512 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.992269    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:35:03.992512 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:03.992318    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:35:04.146645 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:04.090369    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:04.146645 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.094501    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:30:03 +0000 UTC" deadline="2027-12-24 14:45:50.109741771 +0000 UTC"
Jul 06 20:35:04.146645 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.094522    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12858h10m46.015222939s"
Jul 06 20:35:04.337081 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:04.191313    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:04.337081 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:04.291838    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:04.392892    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found"
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.425433    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.444212    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:04.444542    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22a2c57117da9e2036a061b6a5bcbad.slice/crio-f195859aa71d623a049ca59972d920f5d4c502741b85c657598b456a50680565 WatchSource:0}: Error finding container f195859aa71d623a049ca59972d920f5d4c502741b85c657598b456a50680565: Status 404 returned error can't find the container with id f195859aa71d623a049ca59972d920f5d4c502741b85c657598b456a50680565
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.450758    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.470785    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.482587    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.483726    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"
Jul 06 20:35:04.518935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.493560    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:35:04.760290 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:04.760259    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb97d123bad45427ccd732f341cd5be2.slice/crio-38cb14b637c9302dccbd4377ea5d22b0ddd79d1841987a2b9c9fad2edc7f78e7 WatchSource:0}: Error finding container 38cb14b637c9302dccbd4377ea5d22b0ddd79d1841987a2b9c9fad2edc7f78e7: Status 404 returned error can't find the container with id 38cb14b637c9302dccbd4377ea5d22b0ddd79d1841987a2b9c9fad2edc7f78e7
Jul 06 20:35:04.871579 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:04.871553    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:05.036729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.036649    2582 apiserver.go:52] "Watching apiserver"
Jul 06 20:35:05.048066 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.048045    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 20:35:05.051909 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.050714    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-j8bqp","openshift-network-operator/iptables-alerter-w8gzj","openshift-ovn-kubernetes/ovnkube-node-b2hcz","kube-system/global-pull-secret-syncer-w86tz","kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75","openshift-cluster-node-tuning-operator/tuned-jpg75","openshift-dns/node-resolver-hx4xq","openshift-multus/multus-clchg","openshift-network-diagnostics/network-check-target-p5hbx","kube-system/konnectivity-agent-5hpfr","openshift-image-registry/node-ca-4jcp4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal","openshift-multus/multus-additional-cni-plugins-h2pdn"]
Jul 06 20:35:05.053372 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.053349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.055655 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.055490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.056493 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.056477    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:05.057008 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.056990    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:05.057154 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.057126    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dphrb\""
Jul 06 20:35:05.058037 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.057906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.058437 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.058416    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 20:35:05.058437 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.058433    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bgbdp\""
Jul 06 20:35:05.058979 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.058606    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 20:35:05.060362 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.060255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.060362 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.060317    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:05.060518 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.060394    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 20:35:05.060640 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.060624    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 20:35:05.060837 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.060818    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7l8k6\""
Jul 06 20:35:05.062059 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.061764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 20:35:05.062059 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.061791    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 20:35:05.062059 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.061801    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 20:35:05.062059 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.061812    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 20:35:05.062871 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.062850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:05.062977 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.062929    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:05.064705 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.064689    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.067221 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.067204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.067779 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.067758    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 20:35:05.068044 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.068013    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 20:35:05.068044 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.068017    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rxvf9\""
Jul 06 20:35:05.068574 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.068107    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 20:35:05.069556 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.069536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.069644 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.069536    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 20:35:05.070107 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.070085    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 20:35:05.070349 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.070116    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 20:35:05.070349 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.070170    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 20:35:05.070938 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.070919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-66qlw\""
Jul 06 20:35:05.070938 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.070931    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 20:35:05.072247 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.072227    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mx9gf\""
Jul 06 20:35:05.072337 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.072321    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 20:35:05.072506 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.072483    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 20:35:05.072632 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.072618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.074690 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.074671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:05.074809 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.074738    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:05.075277 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.075257    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:05.075277 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.075272    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 20:35:05.075412 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.075295    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:05.075538 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.075525    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47pwf\""
Jul 06 20:35:05.077313 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.076991    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.079435 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.079418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.082309 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.082292    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 20:35:05.082618 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.082560    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 20:35:05.082929 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.082915    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-h9vkc\""
Jul 06 20:35:05.083617 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-os-release\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.083701 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-socket-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.083701 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-registration-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.083701 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-hosts-file\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.083862 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sd6f\" (UniqueName: \"kubernetes.io/projected/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-kube-api-access-9sd6f\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:05.083862 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-cni-multus\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.083862 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-multus-certs\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.083862 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.084132 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-cni-netd\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.084132 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-os-release\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.084132 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-var-lib-kubelet\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.084132 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.083942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.085553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.084885    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 20:35:05.085553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.084888    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 20:35:05.085553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.085553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjgd\" (UniqueName: \"kubernetes.io/projected/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-kube-api-access-9gjgd\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.085769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-ovnkube-config\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.085769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-env-overrides\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.085769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-socket-dir-parent\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.085769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-lib-modules\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.085769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.085769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-kubelet\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.086026 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-cni-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.086026 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085843    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 20:35:05.086026 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2m7\" (UniqueName: \"kubernetes.io/projected/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-kube-api-access-xd2m7\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.086026 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.085989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysconfig\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.086026 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086011    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61bc382a-2fe9-49fe-b533-2c54de55ec41-tmp\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.086225 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/042b84ca-928d-462f-8a34-fb128944ed65-dbus\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.086225 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-system-cni-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.086225 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-systemd\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.087149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-var-lib-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.087149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-log-socket\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.087149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-modprobe-d\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.087149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086668    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588jl\""
Jul 06 20:35:05.087149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-run\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.087149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.087149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.086952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-cni-bin\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f00865b-26d8-420c-8d6e-c47801162baa-multus-daemon-config\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d012d6f-567b-4a22-851b-4009bc11ecfb-host-slash\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8t2r\" (UniqueName: \"kubernetes.io/projected/abb7427f-601e-433f-9703-51470d84332c-kube-api-access-g8t2r\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-k8s-cni-cncf-io\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-device-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-sys\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.087507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-node-log\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-netns\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-kubelet\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysctl-conf\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-run-netns\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-kubernetes\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-systemd\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.087833 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvb2t\" (UniqueName: \"kubernetes.io/projected/3f00865b-26d8-420c-8d6e-c47801162baa-kube-api-access-wvb2t\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-system-cni-dir\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cnibin\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.087983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysctl-d\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2mx\" (UniqueName: \"kubernetes.io/projected/61bc382a-2fe9-49fe-b533-2c54de55ec41-kube-api-access-db2mx\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-ovnkube-script-lib\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f00865b-26d8-420c-8d6e-c47801162baa-cni-binary-copy\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.088192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d012d6f-567b-4a22-851b-4009bc11ecfb-iptables-alerter-script\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47dx7\" (UniqueName: \"kubernetes.io/projected/2d012d6f-567b-4a22-851b-4009bc11ecfb-kube-api-access-47dx7\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/042b84ca-928d-462f-8a34-fb128944ed65-kubelet-config\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-sys-fs\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-tmp-dir\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-ovn\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-host\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-tuned\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.088520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-systemd-units\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-slash\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abb7427f-601e-433f-9703-51470d84332c-ovn-node-metrics-cert\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-cnibin\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-hostroot\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8sq8\" (UniqueName: \"kubernetes.io/projected/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-kube-api-access-g8sq8\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-etc-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-cni-bin\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-conf-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.089027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.088873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-etc-kubernetes\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.090036 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.089948    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:35:05.121914 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.121893    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zl7mf"
Jul 06 20:35:05.130390 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.130365    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zl7mf"
Jul 06 20:35:05.147386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.147362    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:05.171991 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.171929    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 20:35:05.189665 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d012d6f-567b-4a22-851b-4009bc11ecfb-host-slash\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.189816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8t2r\" (UniqueName: \"kubernetes.io/projected/abb7427f-601e-433f-9703-51470d84332c-kube-api-access-g8t2r\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.189816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-k8s-cni-cncf-io\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.189816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189734    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fef030ed-1788-4ced-acbc-6b11fb567dd5-konnectivity-ca\") pod \"konnectivity-agent-5hpfr\" (UID: \"fef030ed-1788-4ced-acbc-6b11fb567dd5\") " pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.189816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jqpn\" (UniqueName: \"kubernetes.io/projected/700e472f-98d4-4970-b9ea-3b1d6ed575e6-kube-api-access-9jqpn\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.189816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-device-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.189816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-sys\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-node-log\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189869    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d012d6f-567b-4a22-851b-4009bc11ecfb-host-slash\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-netns\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189923    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-netns\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-kubelet\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysctl-conf\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-run-netns\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190054    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-device-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-kubernetes\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-systemd\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvb2t\" (UniqueName: \"kubernetes.io/projected/3f00865b-26d8-420c-8d6e-c47801162baa-kube-api-access-wvb2t\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.190143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190138    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-sys\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-system-cni-dir\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-node-log\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.190145    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190187    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-kubelet\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cnibin\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysctl-d\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2mx\" (UniqueName: \"kubernetes.io/projected/61bc382a-2fe9-49fe-b533-2c54de55ec41-kube-api-access-db2mx\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-ovnkube-script-lib\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f00865b-26d8-420c-8d6e-c47801162baa-cni-binary-copy\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190406    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-system-cni-dir\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190437    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-run-netns\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d012d6f-567b-4a22-851b-4009bc11ecfb-iptables-alerter-script\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.190839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190478    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-kubernetes\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47dx7\" (UniqueName: \"kubernetes.io/projected/2d012d6f-567b-4a22-851b-4009bc11ecfb-kube-api-access-47dx7\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190504    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-systemd\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/042b84ca-928d-462f-8a34-fb128944ed65-kubelet-config\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.189981    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-k8s-cni-cncf-io\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190573    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cnibin\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190596    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-sys-fs\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-sys-fs\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.190631    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:05.690593475 +0000 UTC m=+3.024934054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190709    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190734    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysctl-conf\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190770    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190792    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.191550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190816    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysctl-d\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.190933    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/042b84ca-928d-462f-8a34-fb128944ed65-kubelet-config\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-tmp-dir\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-ovn\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700e472f-98d4-4970-b9ea-3b1d6ed575e6-host\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-host\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-tuned\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191180    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d012d6f-567b-4a22-851b-4009bc11ecfb-iptables-alerter-script\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-systemd-units\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191247    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-systemd-units\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191251    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-host\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-slash\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191347    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-ovnkube-script-lib\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abb7427f-601e-433f-9703-51470d84332c-ovn-node-metrics-cert\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-cnibin\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191392    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f00865b-26d8-420c-8d6e-c47801162baa-cni-binary-copy\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-hostroot\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191395    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-slash\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8sq8\" (UniqueName: \"kubernetes.io/projected/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-kube-api-access-g8sq8\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191451    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191459    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-etc-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-ovn\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-cni-bin\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191519    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-hostroot\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-conf-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191548    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-cnibin\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.192860 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-etc-kubernetes\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-os-release\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191619    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-conf-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191619    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-etc-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191546    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191650    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-tmp-dir\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191663    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-cni-bin\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-socket-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-registration-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191738    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-etc-kubernetes\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191743    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-os-release\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-hosts-file\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191781    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-registration-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sd6f\" (UniqueName: \"kubernetes.io/projected/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-kube-api-access-9sd6f\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:05.193569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191811    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-hosts-file\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191814    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-socket-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-cni-multus\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-multus-certs\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.191997    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-cni-multus\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-cni-netd\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192139    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-run-multus-certs\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-os-release\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192185    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-cni-netd\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/700e472f-98d4-4970-b9ea-3b1d6ed575e6-serviceca\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-var-lib-kubelet\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192254    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-os-release\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192300    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-var-lib-kubelet\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjgd\" (UniqueName: \"kubernetes.io/projected/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-kube-api-access-9gjgd\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-ovnkube-config\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-env-overrides\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.192382    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-socket-dir-parent\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-lib-modules\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.192428    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret podName:042b84ca-928d-462f-8a34-fb128944ed65 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:05.692413695 +0000 UTC m=+3.026754283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret") pod "global-pull-secret-syncer-w86tz" (UID: "042b84ca-928d-462f-8a34-fb128944ed65") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192466    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-kubelet\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-cni-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192521    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-lib-modules\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.194927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2m7\" (UniqueName: \"kubernetes.io/projected/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-kube-api-access-xd2m7\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192567    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192677    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-host-kubelet\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192812    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-cni-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192862    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-multus-socket-dir-parent\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fef030ed-1788-4ced-acbc-6b11fb567dd5-agent-certs\") pod \"konnectivity-agent-5hpfr\" (UID: \"fef030ed-1788-4ced-acbc-6b11fb567dd5\") " pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysconfig\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.192991    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-ovnkube-config\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61bc382a-2fe9-49fe-b533-2c54de55ec41-tmp\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/042b84ca-928d-462f-8a34-fb128944ed65-dbus\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-system-cni-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193098    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abb7427f-601e-433f-9703-51470d84332c-env-overrides\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-systemd\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193161    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-system-cni-dir\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.195577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-var-lib-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193196    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-var-lib-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-log-socket\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-modprobe-d\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193238    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-log-socket\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193287    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-systemd\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193296    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/042b84ca-928d-462f-8a34-fb128944ed65-dbus\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193338    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-sysconfig\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193360    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-modprobe-d\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-run\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-cni-bin\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193451    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61bc382a-2fe9-49fe-b533-2c54de55ec41-run\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f00865b-26d8-420c-8d6e-c47801162baa-multus-daemon-config\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193594    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f00865b-26d8-420c-8d6e-c47801162baa-host-var-lib-cni-bin\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.196283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193617    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abb7427f-601e-433f-9703-51470d84332c-run-openvswitch\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.197034 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.193943    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f00865b-26d8-420c-8d6e-c47801162baa-multus-daemon-config\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.197187 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.197167    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61bc382a-2fe9-49fe-b533-2c54de55ec41-tmp\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.199563 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.199539    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abb7427f-601e-433f-9703-51470d84332c-ovn-node-metrics-cert\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.199907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.199868    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61bc382a-2fe9-49fe-b533-2c54de55ec41-etc-tuned\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.209242 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.209216    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47dx7\" (UniqueName: \"kubernetes.io/projected/2d012d6f-567b-4a22-851b-4009bc11ecfb-kube-api-access-47dx7\") pod \"iptables-alerter-w8gzj\" (UID: \"2d012d6f-567b-4a22-851b-4009bc11ecfb\") " pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.209352 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.209317    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2mx\" (UniqueName: \"kubernetes.io/projected/61bc382a-2fe9-49fe-b533-2c54de55ec41-kube-api-access-db2mx\") pod \"tuned-jpg75\" (UID: \"61bc382a-2fe9-49fe-b533-2c54de55ec41\") " pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.210262 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.210220    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8sq8\" (UniqueName: \"kubernetes.io/projected/09d366b1-0bfa-49f1-8cfe-d68f9fd5dece-kube-api-access-g8sq8\") pod \"node-resolver-hx4xq\" (UID: \"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece\") " pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.211428 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.211357    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sd6f\" (UniqueName: \"kubernetes.io/projected/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-kube-api-access-9sd6f\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:05.213594 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.212807    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvb2t\" (UniqueName: \"kubernetes.io/projected/3f00865b-26d8-420c-8d6e-c47801162baa-kube-api-access-wvb2t\") pod \"multus-clchg\" (UID: \"3f00865b-26d8-420c-8d6e-c47801162baa\") " pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.213594 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.213109    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8t2r\" (UniqueName: \"kubernetes.io/projected/abb7427f-601e-433f-9703-51470d84332c-kube-api-access-g8t2r\") pod \"ovnkube-node-b2hcz\" (UID: \"abb7427f-601e-433f-9703-51470d84332c\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.213594 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.213339    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2m7\" (UniqueName: \"kubernetes.io/projected/1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4-kube-api-access-xd2m7\") pod \"multus-additional-cni-plugins-h2pdn\" (UID: \"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4\") " pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.215047 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.214951    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjgd\" (UniqueName: \"kubernetes.io/projected/9f877fcc-c4dd-4a67-a14b-8f532e6b1814-kube-api-access-9gjgd\") pod \"aws-ebs-csi-driver-node-hgb75\" (UID: \"9f877fcc-c4dd-4a67-a14b-8f532e6b1814\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.294678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.294588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fef030ed-1788-4ced-acbc-6b11fb567dd5-konnectivity-ca\") pod \"konnectivity-agent-5hpfr\" (UID: \"fef030ed-1788-4ced-acbc-6b11fb567dd5\") " pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.294678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.294634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqpn\" (UniqueName: \"kubernetes.io/projected/700e472f-98d4-4970-b9ea-3b1d6ed575e6-kube-api-access-9jqpn\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.294882 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.294686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700e472f-98d4-4970-b9ea-3b1d6ed575e6-host\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.294882 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.294718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:05.294882 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.294746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/700e472f-98d4-4970-b9ea-3b1d6ed575e6-serviceca\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.294882 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.294784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fef030ed-1788-4ced-acbc-6b11fb567dd5-agent-certs\") pod \"konnectivity-agent-5hpfr\" (UID: \"fef030ed-1788-4ced-acbc-6b11fb567dd5\") " pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.296638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.295284    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fef030ed-1788-4ced-acbc-6b11fb567dd5-konnectivity-ca\") pod \"konnectivity-agent-5hpfr\" (UID: \"fef030ed-1788-4ced-acbc-6b11fb567dd5\") " pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.296638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.295380    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700e472f-98d4-4970-b9ea-3b1d6ed575e6-host\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.296638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.296599    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/700e472f-98d4-4970-b9ea-3b1d6ed575e6-serviceca\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.299348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.299302    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fef030ed-1788-4ced-acbc-6b11fb567dd5-agent-certs\") pod \"konnectivity-agent-5hpfr\" (UID: \"fef030ed-1788-4ced-acbc-6b11fb567dd5\") " pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.302800 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.302519    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:05.302800 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.302543    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:05.302800 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.302557    2582 projected.go:196] Error preparing data for projected volume kube-api-access-88bph for pod openshift-network-diagnostics/network-check-target-p5hbx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:05.302800 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.302631    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph podName:78929ea1-2b61-4d14-9dba-1890eeb9e9b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:05.802613338 +0000 UTC m=+3.136953921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-88bph" (UniqueName: "kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph") pod "network-check-target-p5hbx" (UID: "78929ea1-2b61-4d14-9dba-1890eeb9e9b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:05.306024 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.305951    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqpn\" (UniqueName: \"kubernetes.io/projected/700e472f-98d4-4970-b9ea-3b1d6ed575e6-kube-api-access-9jqpn\") pod \"node-ca-4jcp4\" (UID: \"700e472f-98d4-4970-b9ea-3b1d6ed575e6\") " pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.337706 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.337623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" event={"ID":"bb97d123bad45427ccd732f341cd5be2","Type":"ContainerStarted","Data":"38cb14b637c9302dccbd4377ea5d22b0ddd79d1841987a2b9c9fad2edc7f78e7"}
Jul 06 20:35:05.339003 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.338977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" event={"ID":"f22a2c57117da9e2036a061b6a5bcbad","Type":"ContainerStarted","Data":"f195859aa71d623a049ca59972d920f5d4c502741b85c657598b456a50680565"}
Jul 06 20:35:05.366562 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.366530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpg75"
Jul 06 20:35:05.374479 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.374455    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hx4xq"
Jul 06 20:35:05.376500 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.376471    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bc382a_2fe9_49fe_b533_2c54de55ec41.slice/crio-989123b1494ae804c8201000f31eeb796f132c274fdb864e4eaa3b2679974f03 WatchSource:0}: Error finding container 989123b1494ae804c8201000f31eeb796f132c274fdb864e4eaa3b2679974f03: Status 404 returned error can't find the container with id 989123b1494ae804c8201000f31eeb796f132c274fdb864e4eaa3b2679974f03
Jul 06 20:35:05.382130 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.382104    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:05.385388 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.385359    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d366b1_0bfa_49f1_8cfe_d68f9fd5dece.slice/crio-29d32db4e92ed3fb018febce7d2c51953f4217fcd35fe25461052a0a96f8c680 WatchSource:0}: Error finding container 29d32db4e92ed3fb018febce7d2c51953f4217fcd35fe25461052a0a96f8c680: Status 404 returned error can't find the container with id 29d32db4e92ed3fb018febce7d2c51953f4217fcd35fe25461052a0a96f8c680
Jul 06 20:35:05.391444 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.391260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75"
Jul 06 20:35:05.392399 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.392374    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb7427f_601e_433f_9703_51470d84332c.slice/crio-5f22c34856c48242542518b16a7b70136d8551932f56c86bc48cddecd614a0a5 WatchSource:0}: Error finding container 5f22c34856c48242542518b16a7b70136d8551932f56c86bc48cddecd614a0a5: Status 404 returned error can't find the container with id 5f22c34856c48242542518b16a7b70136d8551932f56c86bc48cddecd614a0a5
Jul 06 20:35:05.396417 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.396394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-clchg"
Jul 06 20:35:05.401729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.401709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2pdn"
Jul 06 20:35:05.402052 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.402026    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f877fcc_c4dd_4a67_a14b_8f532e6b1814.slice/crio-88eecf9c0273929444b536c13a17e015982c5447b5cfeca31aa7614e4caf78fe WatchSource:0}: Error finding container 88eecf9c0273929444b536c13a17e015982c5447b5cfeca31aa7614e4caf78fe: Status 404 returned error can't find the container with id 88eecf9c0273929444b536c13a17e015982c5447b5cfeca31aa7614e4caf78fe
Jul 06 20:35:05.406612 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.406576    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f00865b_26d8_420c_8d6e_c47801162baa.slice/crio-bd3828976fc0c85bd09a97e35fc8e3cea84c18f7d670dfaf451272aae239cc0c WatchSource:0}: Error finding container bd3828976fc0c85bd09a97e35fc8e3cea84c18f7d670dfaf451272aae239cc0c: Status 404 returned error can't find the container with id bd3828976fc0c85bd09a97e35fc8e3cea84c18f7d670dfaf451272aae239cc0c
Jul 06 20:35:05.410059 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.409941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w8gzj"
Jul 06 20:35:05.412661 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.412638    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcdaa3b_12f9_4d32_8510_b79a2be6e2e4.slice/crio-e22be99228968599f60013c92eb3b975025ca9cc10b06c0d41db9664b1571dc3 WatchSource:0}: Error finding container e22be99228968599f60013c92eb3b975025ca9cc10b06c0d41db9664b1571dc3: Status 404 returned error can't find the container with id e22be99228968599f60013c92eb3b975025ca9cc10b06c0d41db9664b1571dc3
Jul 06 20:35:05.417086 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.417066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:05.421407 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.421385    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d012d6f_567b_4a22_851b_4009bc11ecfb.slice/crio-8b629eb26d2ff06dfa8ae4481bcf616d71859080d5636a3255b3f56dfd9c42a7 WatchSource:0}: Error finding container 8b629eb26d2ff06dfa8ae4481bcf616d71859080d5636a3255b3f56dfd9c42a7: Status 404 returned error can't find the container with id 8b629eb26d2ff06dfa8ae4481bcf616d71859080d5636a3255b3f56dfd9c42a7
Jul 06 20:35:05.421502 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.421482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4jcp4"
Jul 06 20:35:05.434482 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.434432    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef030ed_1788_4ced_acbc_6b11fb567dd5.slice/crio-b986ddfa7cb50e22c3b56ecbc863034cc41ae139864efa151d6e54b38cf6c092 WatchSource:0}: Error finding container b986ddfa7cb50e22c3b56ecbc863034cc41ae139864efa151d6e54b38cf6c092: Status 404 returned error can't find the container with id b986ddfa7cb50e22c3b56ecbc863034cc41ae139864efa151d6e54b38cf6c092
Jul 06 20:35:05.437113 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:05.437093    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700e472f_98d4_4970_b9ea_3b1d6ed575e6.slice/crio-6ec1f53103edece6c6efd4dd383d5de95aa8f6514aaccc8accd63062fb334427 WatchSource:0}: Error finding container 6ec1f53103edece6c6efd4dd383d5de95aa8f6514aaccc8accd63062fb334427: Status 404 returned error can't find the container with id 6ec1f53103edece6c6efd4dd383d5de95aa8f6514aaccc8accd63062fb334427
Jul 06 20:35:05.697498 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.697377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:05.697498 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.697443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:05.697712 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.697508    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:05.697712 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.697579    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:05.697712 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.697590    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret podName:042b84ca-928d-462f-8a34-fb128944ed65 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:06.697569721 +0000 UTC m=+4.031910310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret") pod "global-pull-secret-syncer-w86tz" (UID: "042b84ca-928d-462f-8a34-fb128944ed65") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:05.697712 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.697628    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:06.697613652 +0000 UTC m=+4.031954239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:05.898760 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:05.898720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:05.898985 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.898880    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:05.898985 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.898902    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:05.898985 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.898912    2582 projected.go:196] Error preparing data for projected volume kube-api-access-88bph for pod openshift-network-diagnostics/network-check-target-p5hbx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:05.898985 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:05.898979    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph podName:78929ea1-2b61-4d14-9dba-1890eeb9e9b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:06.898949636 +0000 UTC m=+4.233290210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-88bph" (UniqueName: "kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph") pod "network-check-target-p5hbx" (UID: "78929ea1-2b61-4d14-9dba-1890eeb9e9b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:06.133877 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.133737    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:05 +0000 UTC" deadline="2028-02-06 20:37:07.763508675 +0000 UTC"
Jul 06 20:35:06.133877 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.133782    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13920h2m1.629731611s"
Jul 06 20:35:06.331946 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.331178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:06.331946 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.331296    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:06.331946 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.331677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:06.331946 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.331779    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:06.331946 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.331831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:06.331946 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.331899    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:06.349628 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.349595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4jcp4" event={"ID":"700e472f-98d4-4970-b9ea-3b1d6ed575e6","Type":"ContainerStarted","Data":"6ec1f53103edece6c6efd4dd383d5de95aa8f6514aaccc8accd63062fb334427"}
Jul 06 20:35:06.354477 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.354451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5hpfr" event={"ID":"fef030ed-1788-4ced-acbc-6b11fb567dd5","Type":"ContainerStarted","Data":"b986ddfa7cb50e22c3b56ecbc863034cc41ae139864efa151d6e54b38cf6c092"}
Jul 06 20:35:06.368386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.368356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerStarted","Data":"e22be99228968599f60013c92eb3b975025ca9cc10b06c0d41db9664b1571dc3"}
Jul 06 20:35:06.371092 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.371017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-clchg" event={"ID":"3f00865b-26d8-420c-8d6e-c47801162baa","Type":"ContainerStarted","Data":"bd3828976fc0c85bd09a97e35fc8e3cea84c18f7d670dfaf451272aae239cc0c"}
Jul 06 20:35:06.373364 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.373278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75" event={"ID":"9f877fcc-c4dd-4a67-a14b-8f532e6b1814","Type":"ContainerStarted","Data":"88eecf9c0273929444b536c13a17e015982c5447b5cfeca31aa7614e4caf78fe"}
Jul 06 20:35:06.388609 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.388540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"5f22c34856c48242542518b16a7b70136d8551932f56c86bc48cddecd614a0a5"}
Jul 06 20:35:06.401703 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.401674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hx4xq" event={"ID":"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece","Type":"ContainerStarted","Data":"29d32db4e92ed3fb018febce7d2c51953f4217fcd35fe25461052a0a96f8c680"}
Jul 06 20:35:06.404534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.404507    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w8gzj" event={"ID":"2d012d6f-567b-4a22-851b-4009bc11ecfb","Type":"ContainerStarted","Data":"8b629eb26d2ff06dfa8ae4481bcf616d71859080d5636a3255b3f56dfd9c42a7"}
Jul 06 20:35:06.418704 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.418655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpg75" event={"ID":"61bc382a-2fe9-49fe-b533-2c54de55ec41","Type":"ContainerStarted","Data":"989123b1494ae804c8201000f31eeb796f132c274fdb864e4eaa3b2679974f03"}
Jul 06 20:35:06.707690 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.707596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:06.707690 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.707667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:06.707902 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.707801    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:06.707902 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.707861    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:08.70784453 +0000 UTC m=+6.042185107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:06.708436 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.708246    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:06.708436 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.708295    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret podName:042b84ca-928d-462f-8a34-fb128944ed65 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:08.708281617 +0000 UTC m=+6.042622192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret") pod "global-pull-secret-syncer-w86tz" (UID: "042b84ca-928d-462f-8a34-fb128944ed65") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:06.910390 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:06.909711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:06.910390 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.909917    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:06.910390 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.909940    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:06.910390 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.909955    2582 projected.go:196] Error preparing data for projected volume kube-api-access-88bph for pod openshift-network-diagnostics/network-check-target-p5hbx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:06.910390 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:06.910030    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph podName:78929ea1-2b61-4d14-9dba-1890eeb9e9b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:08.910012385 +0000 UTC m=+6.244352977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-88bph" (UniqueName: "kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph") pod "network-check-target-p5hbx" (UID: "78929ea1-2b61-4d14-9dba-1890eeb9e9b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:07.134599 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:07.134509    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:05 +0000 UTC" deadline="2028-04-03 11:05:15.607697128 +0000 UTC"
Jul 06 20:35:07.134599 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:07.134553    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15278h30m8.473148985s"
Jul 06 20:35:08.331085 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:08.331048    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:08.331085 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:08.331082    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:08.331600 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:08.331048    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:08.331600 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.331207    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:08.331600 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.331270    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:08.331600 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.331347    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:08.727795 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:08.727706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:08.727795 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:08.727773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:08.728069 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.727926    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:08.728069 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.728002    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:12.727983255 +0000 UTC m=+10.062323833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:08.728397 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.728379    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:08.728473 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.728430    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret podName:042b84ca-928d-462f-8a34-fb128944ed65 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:12.728415593 +0000 UTC m=+10.062756184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret") pod "global-pull-secret-syncer-w86tz" (UID: "042b84ca-928d-462f-8a34-fb128944ed65") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:08.929901 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:08.929862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:08.930113 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.930088    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:08.930113 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.930113    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:08.930230 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.930126    2582 projected.go:196] Error preparing data for projected volume kube-api-access-88bph for pod openshift-network-diagnostics/network-check-target-p5hbx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:08.930230 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:08.930194    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph podName:78929ea1-2b61-4d14-9dba-1890eeb9e9b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:12.930176698 +0000 UTC m=+10.264517272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-88bph" (UniqueName: "kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph") pod "network-check-target-p5hbx" (UID: "78929ea1-2b61-4d14-9dba-1890eeb9e9b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:10.330388 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:10.330358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:10.330761 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:10.330489    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:10.330869 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:10.330850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:10.330994 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:10.330850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:10.331066 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:10.330988    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:10.331122 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:10.331102    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:12.330664 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:12.330630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:12.330664 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:12.330650    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:12.331071 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:12.330733    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:12.331071 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.330747    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:12.331071 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.330807    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:12.331071 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.330857    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:12.756896 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:12.756812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:12.756896 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:12.756873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:12.757130 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.756990    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:12.757130 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.756993    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:12.757130 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.757052    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:20.757032913 +0000 UTC m=+18.091373500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:12.757130 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.757071    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret podName:042b84ca-928d-462f-8a34-fb128944ed65 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:20.757062616 +0000 UTC m=+18.091403194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret") pod "global-pull-secret-syncer-w86tz" (UID: "042b84ca-928d-462f-8a34-fb128944ed65") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:12.958647 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:12.958605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:12.958835 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.958790    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:12.958835 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.958818    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:12.958835 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.958833    2582 projected.go:196] Error preparing data for projected volume kube-api-access-88bph for pod openshift-network-diagnostics/network-check-target-p5hbx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:12.958989 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:12.958895    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph podName:78929ea1-2b61-4d14-9dba-1890eeb9e9b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:20.958876692 +0000 UTC m=+18.293217283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-88bph" (UniqueName: "kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph") pod "network-check-target-p5hbx" (UID: "78929ea1-2b61-4d14-9dba-1890eeb9e9b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:14.331067 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:14.330978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:14.331067 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:14.331004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:14.331533 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:14.331109    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:14.331533 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:14.331159    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:14.331533 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:14.331218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:14.331533 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:14.331333    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:16.331422 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:16.330775    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:16.331422 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:16.330826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:16.331422 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:16.331154    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:16.331422 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:16.331261    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:16.331422 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:16.330826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:16.331422 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:16.331387    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:16.460405 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:16.460366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" event={"ID":"f22a2c57117da9e2036a061b6a5bcbad","Type":"ContainerStarted","Data":"639401cba2cc325e292649c888a4d3409650f8050a70a6923962035e767cc858"}
Jul 06 20:35:16.462327 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:16.462292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpg75" event={"ID":"61bc382a-2fe9-49fe-b533-2c54de55ec41","Type":"ContainerStarted","Data":"42b72032c53f8b9ad800890e4e2a6be8e6714530cd63fe5422d5ba6b11e3a4b0"}
Jul 06 20:35:16.480720 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:16.480639    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" podStartSLOduration=12.480621428 podStartE2EDuration="12.480621428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:16.480492108 +0000 UTC m=+13.814832704" watchObservedRunningTime="2026-07-06 20:35:16.480621428 +0000 UTC m=+13.814962025"
Jul 06 20:35:16.501685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:16.501632    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jpg75" podStartSLOduration=3.151694167 podStartE2EDuration="13.501611278s" totalImagesPullingTime="10.349917111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:16.501148583 +0000 UTC m=+13.835489200" watchObservedRunningTime="2026-07-06 20:35:16.501611278 +0000 UTC m=+13.835951873"
Jul 06 20:35:17.467299 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.467263    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerStarted","Data":"c8573b6af8272a9753a14076a8ec9ce5bdc2ffd22f83c58a7a548429331db771"}
Jul 06 20:35:17.469910 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.469875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75" event={"ID":"9f877fcc-c4dd-4a67-a14b-8f532e6b1814","Type":"ContainerStarted","Data":"431f94730af45e5bd278ad209637725a07844131a8a85ad4db22c53d60463305"}
Jul 06 20:35:17.474579 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.474546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hx4xq" event={"ID":"09d366b1-0bfa-49f1-8cfe-d68f9fd5dece","Type":"ContainerStarted","Data":"2bc223220b9f2d9da8263974be28b278849db36cfbf9a3abe6deccdc790d0996"}
Jul 06 20:35:17.476592 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.476565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" event={"ID":"bb97d123bad45427ccd732f341cd5be2","Type":"ContainerStarted","Data":"65dfc2fd5f4c3943b4f7da0209bed7a54d4378d39c1d1e93406c76b3dfb1bf3e"}
Jul 06 20:35:17.478462 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.478414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w8gzj" event={"ID":"2d012d6f-567b-4a22-851b-4009bc11ecfb","Type":"ContainerStarted","Data":"f72a816692ff6d0e68595f1e42b8e8df7d8efde326d57168cd200b9b9fd4098b"}
Jul 06 20:35:17.480178 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.480138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4jcp4" event={"ID":"700e472f-98d4-4970-b9ea-3b1d6ed575e6","Type":"ContainerStarted","Data":"180c8bf1e00ec7d048cf19a44ea019be1b4d7bcd6c64aa490a9b28e6d6f25fd6"}
Jul 06 20:35:17.481736 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.481714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5hpfr" event={"ID":"fef030ed-1788-4ced-acbc-6b11fb567dd5","Type":"ContainerStarted","Data":"3394c32ba394809466e068bf2d2e02dce47b78fa0583218103a9dc21baac34c8"}
Jul 06 20:35:17.539578 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.539516    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hx4xq" podStartSLOduration=4.180022877 podStartE2EDuration="14.53949834s" totalImagesPullingTime="10.359475463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:17.539254378 +0000 UTC m=+14.873594973" watchObservedRunningTime="2026-07-06 20:35:17.53949834 +0000 UTC m=+14.873838935"
Jul 06 20:35:17.581409 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.581330    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w8gzj" podStartSLOduration=4.277544994 podStartE2EDuration="14.58130778s" totalImagesPullingTime="10.303762786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:17.559128042 +0000 UTC m=+14.893468657" watchObservedRunningTime="2026-07-06 20:35:17.58130778 +0000 UTC m=+14.915648377"
Jul 06 20:35:17.601081 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.600793    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5hpfr" podStartSLOduration=4.349363435 podStartE2EDuration="14.600777847s" totalImagesPullingTime="10.251414412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:17.581018579 +0000 UTC m=+14.915359173" watchObservedRunningTime="2026-07-06 20:35:17.600777847 +0000 UTC m=+14.935118443"
Jul 06 20:35:17.601235 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:17.601115    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4jcp4" podStartSLOduration=4.3098005090000004 podStartE2EDuration="14.601100795s" totalImagesPullingTime="10.291300286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:17.600772642 +0000 UTC m=+14.935113237" watchObservedRunningTime="2026-07-06 20:35:17.601100795 +0000 UTC m=+14.935441398"
Jul 06 20:35:18.330359 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:18.330322    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:18.330540 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:18.330372    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:18.330540 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:18.330418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:18.330540 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:18.330433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:18.330540 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:18.330528    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:18.330731 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:18.330626    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:20.009148 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.009109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:20.009761 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.009740    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:20.331120 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.331041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:20.331120 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.331085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:20.331299 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.331041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:20.331299 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:20.331152    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:20.331299 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:20.331228    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:20.331429 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:20.331343    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:20.488944 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.488912    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4" containerID="c8573b6af8272a9753a14076a8ec9ce5bdc2ffd22f83c58a7a548429331db771" exitCode=0
Jul 06 20:35:20.489107 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.488999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerDied","Data":"c8573b6af8272a9753a14076a8ec9ce5bdc2ffd22f83c58a7a548429331db771"}
Jul 06 20:35:20.490958 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.490941    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb97d123bad45427ccd732f341cd5be2" containerID="65dfc2fd5f4c3943b4f7da0209bed7a54d4378d39c1d1e93406c76b3dfb1bf3e" exitCode=0
Jul 06 20:35:20.491054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.490994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" event={"ID":"bb97d123bad45427ccd732f341cd5be2","Type":"ContainerDied","Data":"65dfc2fd5f4c3943b4f7da0209bed7a54d4378d39c1d1e93406c76b3dfb1bf3e"}
Jul 06 20:35:20.491172 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.491151    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:20.491856 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.491759    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5hpfr"
Jul 06 20:35:20.821590 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.821547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:20.821754 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:20.821602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:20.821754 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:20.821714    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:20.821754 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:20.821725    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:20.821921 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:20.821759    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:36.8217447 +0000 UTC m=+34.156085273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:20.821921 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:20.821784    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret podName:042b84ca-928d-462f-8a34-fb128944ed65 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:36.821765141 +0000 UTC m=+34.156105717 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret") pod "global-pull-secret-syncer-w86tz" (UID: "042b84ca-928d-462f-8a34-fb128944ed65") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:21.022758 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:21.022717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:21.023336 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:21.022910    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:21.023336 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:21.022939    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:21.023336 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:21.022955    2582 projected.go:196] Error preparing data for projected volume kube-api-access-88bph for pod openshift-network-diagnostics/network-check-target-p5hbx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:21.023336 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:21.023034    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph podName:78929ea1-2b61-4d14-9dba-1890eeb9e9b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:37.023018367 +0000 UTC m=+34.357358942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-88bph" (UniqueName: "kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph") pod "network-check-target-p5hbx" (UID: "78929ea1-2b61-4d14-9dba-1890eeb9e9b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:22.330630 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:22.330594    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:22.331211 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:22.330704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:22.331211 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:22.330712    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:22.331211 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:22.330812    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:22.331211 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:22.330844    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:22.331211 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:22.330922    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:24.330568 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:24.330528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:24.331049 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:24.330529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:24.331049 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:24.330678    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:24.331049 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:24.330721    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:24.331049 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:24.330529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:24.331049 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:24.330800    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:25.414647 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.414398    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 20:35:25.502570 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.502534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-clchg" event={"ID":"3f00865b-26d8-420c-8d6e-c47801162baa","Type":"ContainerStarted","Data":"54ce8a6d0a6f5bb1864e9cdd366e4879d9676a9f194b380bb17b81ec84dff09b"}
Jul 06 20:35:25.504193 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.504174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75" event={"ID":"9f877fcc-c4dd-4a67-a14b-8f532e6b1814","Type":"ContainerStarted","Data":"593fb4b5a6a4996755eacb2483822e9e12be96daa6c18f92b21dbecd4650d61d"}
Jul 06 20:35:25.505918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.505895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"907c30c63844a885cb696607392236f35905ebe9fe1df358b643faed1b3f014c"}
Jul 06 20:35:25.506009 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.505958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"d5fec0a1e9f7bf2dc7fbef2afdea06624b97e89b63c29d192fd76a97e625dd49"}
Jul 06 20:35:25.506054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.506034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"380ac4e590baf5e818f296d60662fe20194cabe8cda789a1f2cfc123d38212bc"}
Jul 06 20:35:25.507691 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.507671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" event={"ID":"bb97d123bad45427ccd732f341cd5be2","Type":"ContainerStarted","Data":"30eb25aa76bd11efffa8074a83c9a9946aef5fdfe1cfb3500ef0c8da86638ae1"}
Jul 06 20:35:25.550594 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.550551    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-clchg" podStartSLOduration=2.884272127 podStartE2EDuration="22.550533648s" totalImagesPullingTime="19.666261521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:25.52658443 +0000 UTC m=+22.860925026" watchObservedRunningTime="2026-07-06 20:35:25.550533648 +0000 UTC m=+22.884874244"
Jul 06 20:35:25.550779 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.550751    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" podStartSLOduration=21.550744739 podStartE2EDuration="21.550744739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:25.549994339 +0000 UTC m=+22.884334936" watchObservedRunningTime="2026-07-06 20:35:25.550744739 +0000 UTC m=+22.885085334"
Jul 06 20:35:25.681348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.681320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:25.790143 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:25.790110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:26.147641 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.147344    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:35:25.414645178Z","UUID":"e4c683c2-dbc2-450f-ac21-775d9ccd6b69","Handler":null,"Name":"","Endpoint":""}
Jul 06 20:35:26.149321 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.149298    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 20:35:26.149451 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.149337    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 20:35:26.330589 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.330554    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:26.330589 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.330585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:26.330853 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.330673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:26.330853 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:26.330687    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:26.330853 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:26.330771    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:26.330853 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:26.330828    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:26.513362 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.513321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"0af09a0d568414d1be6d3db8e9deb0a64bb5fcd5fe47240442f0588aa2e7b75c"}
Jul 06 20:35:26.513814 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.513416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"3e74bda8e574573278e551c35fb3af18e4419703dcc265d3eb8cd5657b229866"}
Jul 06 20:35:26.513814 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.513466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"a027fefd04a50c126a1900063ecc2fc85c48b727100e97fedf116d819fa2a415"}
Jul 06 20:35:26.571187 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.571157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:26.932025 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:26.931972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:27.075934 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:27.075884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:27.201519 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:27.201413    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:27.299765 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:27.299733    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:27.457452 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:27.457361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:27.517714 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:27.517684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75" event={"ID":"9f877fcc-c4dd-4a67-a14b-8f532e6b1814","Type":"ContainerStarted","Data":"20cb7600c3f1341a526735ba54595c0033ad8375d43d89b36eab5fcd02c0c73a"}
Jul 06 20:35:27.543165 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:27.543111    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgb75" podStartSLOduration=5.283026734 podStartE2EDuration="24.543096375s" totalImagesPullingTime="19.260069641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:27.542828987 +0000 UTC m=+24.877169585" watchObservedRunningTime="2026-07-06 20:35:27.543096375 +0000 UTC m=+24.877436969"
Jul 06 20:35:27.566624 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:27.566591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:28.330770 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:28.330730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:28.330770 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:28.330753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:28.330770 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:28.330772    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:28.331116 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:28.330858    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:28.331116 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:28.330917    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:28.331116 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:28.331017    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:29.524444 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:29.524409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"c3b6914ea8831f6744b8f62713f4352e2de3de05271cd9e8488d4fda7d8e952a"}
Jul 06 20:35:29.526164 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:29.526140    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4" containerID="fd0923914ffd55eb6ba130bef2ce366f55354e561afbfd29575939f94a420882" exitCode=0
Jul 06 20:35:29.526279 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:29.526172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerDied","Data":"fd0923914ffd55eb6ba130bef2ce366f55354e561afbfd29575939f94a420882"}
Jul 06 20:35:30.330814 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:30.330777    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:30.331120 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:30.330823    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:30.331120 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:30.330777    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:30.331120 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:30.330904    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:30.331120 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:30.330977    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:30.331120 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:30.331054    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:31.532754 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.532551    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4" containerID="6c906dcb9a429359e93da14a956cb2873afe0bfe6da119eb856194b3abe662ce" exitCode=0
Jul 06 20:35:31.533378 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.532603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerDied","Data":"6c906dcb9a429359e93da14a956cb2873afe0bfe6da119eb856194b3abe662ce"}
Jul 06 20:35:31.536322 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.536301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" event={"ID":"abb7427f-601e-433f-9703-51470d84332c","Type":"ContainerStarted","Data":"4dfd675eaabbb07c6d68450ca48bba8620b937a4ea81f29a3233f7cc54e2dc07"}
Jul 06 20:35:31.536616 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.536585    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:31.536738 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.536649    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:31.552822 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.552797    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:31.552934 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.552903    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:31.598357 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:31.598296    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz" podStartSLOduration=8.917622005 podStartE2EDuration="28.598281029s" totalImagesPullingTime="19.680659024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:31.596829993 +0000 UTC m=+28.931170588" watchObservedRunningTime="2026-07-06 20:35:31.598281029 +0000 UTC m=+28.932621624"
Jul 06 20:35:32.330728 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:32.330692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:32.331026 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:32.330831    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:32.331238 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:32.331216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:32.331333 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:32.331312    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:32.331387 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:32.331366    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:32.331464 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:32.331446    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:32.543181 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:32.543155    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4" containerID="2282d8b48cdcaca9838be3633e41edc50a4f748dd9dff685eac9faf69a0329c0" exitCode=0
Jul 06 20:35:32.543479 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:32.543200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerDied","Data":"2282d8b48cdcaca9838be3633e41edc50a4f748dd9dff685eac9faf69a0329c0"}
Jul 06 20:35:32.543479 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:32.543279    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 06 20:35:33.400150 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:33.400120    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w86tz"]
Jul 06 20:35:33.400301 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:33.400245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:33.400373 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:33.400351    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:33.402568 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:33.402545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p5hbx"]
Jul 06 20:35:33.402673 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:33.402652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:33.402771 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:33.402736    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:33.405208 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:33.404909    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j8bqp"]
Jul 06 20:35:33.405208 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:33.405035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:33.405208 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:33.405133    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:33.545742 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:33.545718    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 06 20:35:35.198693 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:35.198652    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:35:35.330821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:35.330752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:35.330821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:35.330753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:35.330987 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:35.330868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:35.330987 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:35.330874    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:35.330987 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:35.330945    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:35.331091 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:35.331034    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:36.516845 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:36.516613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:36.615527 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:36.615488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:36.837155 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:36.836999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:36.837155 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:36.837072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:36.837155 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:36.837146    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:36.837449 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:36.837189    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:36.837449 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:36.837220    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret podName:042b84ca-928d-462f-8a34-fb128944ed65 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:08.83720157 +0000 UTC m=+66.171542155 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret") pod "global-pull-secret-syncer-w86tz" (UID: "042b84ca-928d-462f-8a34-fb128944ed65") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:36.837449 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:36.837242    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:08.837229874 +0000 UTC m=+66.171570455 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:37.038175 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.038135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:37.038388 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:37.038280    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:37.038388 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:37.038299    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:37.038388 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:37.038322    2582 projected.go:196] Error preparing data for projected volume kube-api-access-88bph for pod openshift-network-diagnostics/network-check-target-p5hbx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:37.038388 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:37.038372    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph podName:78929ea1-2b61-4d14-9dba-1890eeb9e9b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:09.03835789 +0000 UTC m=+66.372698466 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-88bph" (UniqueName: "kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph") pod "network-check-target-p5hbx" (UID: "78929ea1-2b61-4d14-9dba-1890eeb9e9b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:37.186130 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.186099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:37.242689 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.242653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:37.330620 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.330589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:37.330781 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.330589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:37.330781 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:37.330712    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j8bqp" podUID="c0ec3ed5-d7c9-4842-a336-9579edc57bf4"
Jul 06 20:35:37.330781 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.330730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:37.330939 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:37.330800    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w86tz" podUID="042b84ca-928d-462f-8a34-fb128944ed65"
Jul 06 20:35:37.330939 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:37.330914    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5hbx" podUID="78929ea1-2b61-4d14-9dba-1890eeb9e9b2"
Jul 06 20:35:37.954911 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.954885    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeReady"
Jul 06 20:35:37.955352 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:37.955027    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 20:35:38.017612 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.017578    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rps8j"]
Jul 06 20:35:38.036129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.036099    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vr2td"]
Jul 06 20:35:38.036265 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.036225    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.039086 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.039064    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 20:35:38.039212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.039151    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 20:35:38.039212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.039200    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dqcct\""
Jul 06 20:35:38.060156 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.060133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vr2td"]
Jul 06 20:35:38.060282 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.060163    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rps8j"]
Jul 06 20:35:38.060282 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.060240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.062713 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.062694    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 20:35:38.062816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.062692    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 20:35:38.062859 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.062825    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r4sgm\""
Jul 06 20:35:38.063200 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.063183    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 20:35:38.146767 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.146732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6c585-6fc0-45f4-a258-e59b0d234bf9-config-volume\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.146767 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.146770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d6c585-6fc0-45f4-a258-e59b0d234bf9-tmp-dir\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.147040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.146791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wp9\" (UniqueName: \"kubernetes.io/projected/a3c1ec3f-6328-48cc-bc69-d20eaab9e41c-kube-api-access-f8wp9\") pod \"ingress-canary-vr2td\" (UID: \"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c\") " pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.147040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.146896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9jt\" (UniqueName: \"kubernetes.io/projected/05d6c585-6fc0-45f4-a258-e59b0d234bf9-kube-api-access-2n9jt\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.147040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.146929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3c1ec3f-6328-48cc-bc69-d20eaab9e41c-cert\") pod \"ingress-canary-vr2td\" (UID: \"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c\") " pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.147040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.147005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.248029 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.247927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6c585-6fc0-45f4-a258-e59b0d234bf9-config-volume\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.248029 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.247983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d6c585-6fc0-45f4-a258-e59b0d234bf9-tmp-dir\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.248251 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.248106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wp9\" (UniqueName: \"kubernetes.io/projected/a3c1ec3f-6328-48cc-bc69-d20eaab9e41c-kube-api-access-f8wp9\") pod \"ingress-canary-vr2td\" (UID: \"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c\") " pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.248251 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.248172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9jt\" (UniqueName: \"kubernetes.io/projected/05d6c585-6fc0-45f4-a258-e59b0d234bf9-kube-api-access-2n9jt\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.248251 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.248201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3c1ec3f-6328-48cc-bc69-d20eaab9e41c-cert\") pod \"ingress-canary-vr2td\" (UID: \"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c\") " pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.248395 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.248250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.248395 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.248253    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d6c585-6fc0-45f4-a258-e59b0d234bf9-tmp-dir\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.248395 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:38.248370    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:38.248531 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:38.248426    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls podName:05d6c585-6fc0-45f4-a258-e59b0d234bf9 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:38.748406426 +0000 UTC m=+36.082747002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls") pod "dns-default-rps8j" (UID: "05d6c585-6fc0-45f4-a258-e59b0d234bf9") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:38.248587 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.248530    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6c585-6fc0-45f4-a258-e59b0d234bf9-config-volume\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.252357 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.252338    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3c1ec3f-6328-48cc-bc69-d20eaab9e41c-cert\") pod \"ingress-canary-vr2td\" (UID: \"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c\") " pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.257558 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.257537    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9jt\" (UniqueName: \"kubernetes.io/projected/05d6c585-6fc0-45f4-a258-e59b0d234bf9-kube-api-access-2n9jt\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.257729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.257704    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wp9\" (UniqueName: \"kubernetes.io/projected/a3c1ec3f-6328-48cc-bc69-d20eaab9e41c-kube-api-access-f8wp9\") pod \"ingress-canary-vr2td\" (UID: \"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c\") " pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.369073 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.369035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vr2td"
Jul 06 20:35:38.502945 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.502678    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vr2td"]
Jul 06 20:35:38.506060 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:38.506035    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c1ec3f_6328_48cc_bc69_d20eaab9e41c.slice/crio-3f3aca6942f960588792e7604c37a800c114fecc5e4ccd6122088d41f9496e57 WatchSource:0}: Error finding container 3f3aca6942f960588792e7604c37a800c114fecc5e4ccd6122088d41f9496e57: Status 404 returned error can't find the container with id 3f3aca6942f960588792e7604c37a800c114fecc5e4ccd6122088d41f9496e57
Jul 06 20:35:38.560679 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.560647    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4" containerID="bdca076878b9c54e6a3a148ed9acb457f50bf0421a4ce1d57f33750caa929891" exitCode=0
Jul 06 20:35:38.560819 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.560716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerDied","Data":"bdca076878b9c54e6a3a148ed9acb457f50bf0421a4ce1d57f33750caa929891"}
Jul 06 20:35:38.561819 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.561793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vr2td" event={"ID":"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c","Type":"ContainerStarted","Data":"3f3aca6942f960588792e7604c37a800c114fecc5e4ccd6122088d41f9496e57"}
Jul 06 20:35:38.751902 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:38.751860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:38.752083 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:38.752025    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:38.752123 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:38.752091    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls podName:05d6c585-6fc0-45f4-a258-e59b0d234bf9 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:39.752075822 +0000 UTC m=+37.086416395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls") pod "dns-default-rps8j" (UID: "05d6c585-6fc0-45f4-a258-e59b0d234bf9") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:39.331245 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.331211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:35:39.332027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.331254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:35:39.332027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.331211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:35:39.335676 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.335644    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:35:39.335813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.335709    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:35:39.335813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.335766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:35:39.335922 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.335648    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:35:39.335958 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.335947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl572\""
Jul 06 20:35:39.336083 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.336057    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6bf8b\""
Jul 06 20:35:39.568243 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.568212    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4" containerID="74146cd8abcf9497735dd052e015e2b52de521f039c270ca43667a9276c3fbc9" exitCode=0
Jul 06 20:35:39.568412 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.568278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerDied","Data":"74146cd8abcf9497735dd052e015e2b52de521f039c270ca43667a9276c3fbc9"}
Jul 06 20:35:39.761181 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:39.761142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:39.761389 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:39.761303    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:39.761389 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:39.761372    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls podName:05d6c585-6fc0-45f4-a258-e59b0d234bf9 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:41.761353285 +0000 UTC m=+39.095693878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls") pod "dns-default-rps8j" (UID: "05d6c585-6fc0-45f4-a258-e59b0d234bf9") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:40.574488 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:40.574152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" event={"ID":"1dcdaa3b-12f9-4d32-8510-b79a2be6e2e4","Type":"ContainerStarted","Data":"3fa5fd6f09c502d20f77b5bd697eb76ee49b898742bcf4289df859297c6a159e"}
Jul 06 20:35:40.603671 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:40.603628    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h2pdn" podStartSLOduration=11.802870555 podStartE2EDuration="37.603614967s" totalImagesPullingTime="25.800744412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:40.602180119 +0000 UTC m=+37.936520713" watchObservedRunningTime="2026-07-06 20:35:40.603614967 +0000 UTC m=+37.937955561"
Jul 06 20:35:40.730011 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:40.729975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:40.807555 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:40.807513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:41.578374 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:41.578341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vr2td" event={"ID":"a3c1ec3f-6328-48cc-bc69-d20eaab9e41c","Type":"ContainerStarted","Data":"4783a2cba759536bf62ba9adfd8579c97f028103c743496e272a0d318ede1c04"}
Jul 06 20:35:41.596496 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:41.596452    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vr2td" podStartSLOduration=2.543498763 podStartE2EDuration="4.596436495s" totalImagesPullingTime="2.052937732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:41.595882923 +0000 UTC m=+38.930223519" watchObservedRunningTime="2026-07-06 20:35:41.596436495 +0000 UTC m=+38.930777090"
Jul 06 20:35:41.777161 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:41.777115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:41.777314 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:41.777231    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:41.777314 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:41.777283    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls podName:05d6c585-6fc0-45f4-a258-e59b0d234bf9 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:45.777269906 +0000 UTC m=+43.111610480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls") pod "dns-default-rps8j" (UID: "05d6c585-6fc0-45f4-a258-e59b0d234bf9") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:45.805119 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:45.804912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:45.805501 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:45.805067    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:45.805501 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:45.805215    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls podName:05d6c585-6fc0-45f4-a258-e59b0d234bf9 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:53.805198884 +0000 UTC m=+51.139539458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls") pod "dns-default-rps8j" (UID: "05d6c585-6fc0-45f4-a258-e59b0d234bf9") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:48.876403 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:48.876369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:48.934345 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:48.934313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:50.107472 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:50.107442    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:50.169227 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:50.169197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:53.861223 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:53.861182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:35:53.861707 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:53.861356    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:35:53.861707 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:53.861419    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls podName:05d6c585-6fc0-45f4-a258-e59b0d234bf9 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:09.861403215 +0000 UTC m=+67.195743787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls") pod "dns-default-rps8j" (UID: "05d6c585-6fc0-45f4-a258-e59b0d234bf9") : secret "dns-default-metrics-tls" not found
Jul 06 20:35:54.830059 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.830026    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"]
Jul 06 20:35:54.845017 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.844991    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"]
Jul 06 20:35:54.845158 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.845096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"
Jul 06 20:35:54.848364 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.848338    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:54.848470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.848363    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jmn2g\""
Jul 06 20:35:54.848470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.848363    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:54.936298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.936269    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-757d9b65f5-7xf7w"]
Jul 06 20:35:54.956984 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.956941    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-757d9b65f5-7xf7w"]
Jul 06 20:35:54.957126 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.957064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:54.960098 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.960036    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 06 20:35:54.960201 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.960104    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 06 20:35:54.960721 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.960699    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 20:35:54.960986 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.960942    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8tnrg\""
Jul 06 20:35:54.961084 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.960999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 06 20:35:54.961084 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.961018    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 20:35:54.961194 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.961022    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 06 20:35:54.968109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:54.968090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbh2\" (UniqueName: \"kubernetes.io/projected/02624e11-4c9e-4ec1-be35-7ce3a74c65af-kube-api-access-8lbh2\") pod \"volume-data-source-validator-5d456fdbcd-sklbl\" (UID: \"02624e11-4c9e-4ec1-be35-7ce3a74c65af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"
Jul 06 20:35:55.032205 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.032178    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-pn8bl"]
Jul 06 20:35:55.048826 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.048801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.053507 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.053479    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-pn8bl"]
Jul 06 20:35:55.054108 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.054086    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 06 20:35:55.054220 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.054208    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-48j9p\""
Jul 06 20:35:55.054330 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.054314    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 06 20:35:55.055957 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.054871    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:55.055957 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.055010    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:55.060708 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.060688    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 06 20:35:55.068694 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.068674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbh2\" (UniqueName: \"kubernetes.io/projected/02624e11-4c9e-4ec1-be35-7ce3a74c65af-kube-api-access-8lbh2\") pod \"volume-data-source-validator-5d456fdbcd-sklbl\" (UID: \"02624e11-4c9e-4ec1-be35-7ce3a74c65af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"
Jul 06 20:35:55.068788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.068716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-default-certificate\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.068788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.068735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.068788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.068749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8kd\" (UniqueName: \"kubernetes.io/projected/30cb12a1-e55d-432b-b856-258d10383502-kube-api-access-qh8kd\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.068901 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.068793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-stats-auth\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.068901 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.068822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.085790 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.085739    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbh2\" (UniqueName: \"kubernetes.io/projected/02624e11-4c9e-4ec1-be35-7ce3a74c65af-kube-api-access-8lbh2\") pod \"volume-data-source-validator-5d456fdbcd-sklbl\" (UID: \"02624e11-4c9e-4ec1-be35-7ce3a74c65af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"
Jul 06 20:35:55.134373 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.134344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:55.154418 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.154397    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"
Jul 06 20:35:55.169992 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.169974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd23fc80-e745-4494-99c4-4dde8a5772f0-trusted-ca\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.170133 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-default-certificate\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.170133 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.170133 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8kd\" (UniqueName: \"kubernetes.io/projected/30cb12a1-e55d-432b-b856-258d10383502-kube-api-access-qh8kd\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.170133 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-stats-auth\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.170133 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.170322 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:55.170213    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:55.670193392 +0000 UTC m=+53.004533966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:55.170322 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:55.170219    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:55.170322 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170234    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qg7\" (UniqueName: \"kubernetes.io/projected/fd23fc80-e745-4494-99c4-4dde8a5772f0-kube-api-access-57qg7\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.170322 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd23fc80-e745-4494-99c4-4dde8a5772f0-serving-cert\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.170322 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:55.170304    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:55.670278251 +0000 UTC m=+53.004618839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : secret "router-metrics-certs-default" not found
Jul 06 20:35:55.170498 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.170335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd23fc80-e745-4494-99c4-4dde8a5772f0-config\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.172372 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.172349    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-default-certificate\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.172447 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.172372    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-stats-auth\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.181165 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.181145    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8kd\" (UniqueName: \"kubernetes.io/projected/30cb12a1-e55d-432b-b856-258d10383502-kube-api-access-qh8kd\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.236788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.236671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:55.270674 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.270650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd23fc80-e745-4494-99c4-4dde8a5772f0-trusted-ca\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.270784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.270711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qg7\" (UniqueName: \"kubernetes.io/projected/fd23fc80-e745-4494-99c4-4dde8a5772f0-kube-api-access-57qg7\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.270784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.270734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd23fc80-e745-4494-99c4-4dde8a5772f0-serving-cert\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.270784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.270752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd23fc80-e745-4494-99c4-4dde8a5772f0-config\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.271485 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.271460    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd23fc80-e745-4494-99c4-4dde8a5772f0-config\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.271628 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.271609    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd23fc80-e745-4494-99c4-4dde8a5772f0-trusted-ca\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.272987 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.272955    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd23fc80-e745-4494-99c4-4dde8a5772f0-serving-cert\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.279831 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.279808    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qg7\" (UniqueName: \"kubernetes.io/projected/fd23fc80-e745-4494-99c4-4dde8a5772f0-kube-api-access-57qg7\") pod \"console-operator-68b757fb67-pn8bl\" (UID: \"fd23fc80-e745-4494-99c4-4dde8a5772f0\") " pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.295460 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.295438    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl"]
Jul 06 20:35:55.298352 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:55.298331    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02624e11_4c9e_4ec1_be35_7ce3a74c65af.slice/crio-06a7b1219136eb217b49a4243c677792b14f32aafac87ca8126511abb5915222 WatchSource:0}: Error finding container 06a7b1219136eb217b49a4243c677792b14f32aafac87ca8126511abb5915222: Status 404 returned error can't find the container with id 06a7b1219136eb217b49a4243c677792b14f32aafac87ca8126511abb5915222
Jul 06 20:35:55.360940 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.360863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:35:55.515163 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.515025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-pn8bl"]
Jul 06 20:35:55.517460 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:55.517438    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd23fc80_e745_4494_99c4_4dde8a5772f0.slice/crio-993fc2736a199c017a1051864256ad79da2c7fddf5506dc4dd0673da18f953a3 WatchSource:0}: Error finding container 993fc2736a199c017a1051864256ad79da2c7fddf5506dc4dd0673da18f953a3: Status 404 returned error can't find the container with id 993fc2736a199c017a1051864256ad79da2c7fddf5506dc4dd0673da18f953a3
Jul 06 20:35:55.611403 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.611319    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" event={"ID":"fd23fc80-e745-4494-99c4-4dde8a5772f0","Type":"ContainerStarted","Data":"993fc2736a199c017a1051864256ad79da2c7fddf5506dc4dd0673da18f953a3"}
Jul 06 20:35:55.612410 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.612389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl" event={"ID":"02624e11-4c9e-4ec1-be35-7ce3a74c65af","Type":"ContainerStarted","Data":"06a7b1219136eb217b49a4243c677792b14f32aafac87ca8126511abb5915222"}
Jul 06 20:35:55.673789 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.673762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.673918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:55.673853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:55.673982 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:55.673918    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:56.673898154 +0000 UTC m=+54.008238727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:55.673982 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:55.673933    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:55.674065 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:55.673994    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:56.673979505 +0000 UTC m=+54.008320092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : secret "router-metrics-certs-default" not found
Jul 06 20:35:56.681733 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:56.681641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:56.681733 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:56.681703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:56.682311 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:56.681831    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:56.682311 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:56.681910    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.681890536 +0000 UTC m=+56.016231112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : secret "router-metrics-certs-default" not found
Jul 06 20:35:56.682311 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:56.681928    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.681919142 +0000 UTC m=+56.016259716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:58.621225 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.621198    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/0.log"
Jul 06 20:35:58.621666 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.621237    2582 generic.go:356] "Generic (PLEG): container finished" podID="fd23fc80-e745-4494-99c4-4dde8a5772f0" containerID="8a07832cda89a4314a908b88b1afc9e03de11f758c18f05cee5444a9612e3d65" exitCode=255
Jul 06 20:35:58.621666 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.621335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" event={"ID":"fd23fc80-e745-4494-99c4-4dde8a5772f0","Type":"ContainerDied","Data":"8a07832cda89a4314a908b88b1afc9e03de11f758c18f05cee5444a9612e3d65"}
Jul 06 20:35:58.622060 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.621892    2582 scope.go:117] "RemoveContainer" containerID="8a07832cda89a4314a908b88b1afc9e03de11f758c18f05cee5444a9612e3d65"
Jul 06 20:35:58.626169 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.626144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl" event={"ID":"02624e11-4c9e-4ec1-be35-7ce3a74c65af","Type":"ContainerStarted","Data":"34879b535429ba3acd04be83d153f5ef4fe39277ab38360ffba975fe1b6b93b4"}
Jul 06 20:35:58.633821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.633802    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"]
Jul 06 20:35:58.658320 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.658270    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-sklbl" podStartSLOduration=2.133424613 podStartE2EDuration="4.658255206s" totalImagesPullingTime="2.524830593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:58.65768222 +0000 UTC m=+55.992022814" watchObservedRunningTime="2026-07-06 20:35:58.658255206 +0000 UTC m=+55.992595780"
Jul 06 20:35:58.666291 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.666270    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"]
Jul 06 20:35:58.666401 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.666370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"
Jul 06 20:35:58.669175 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.669158    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-d5xr5\""
Jul 06 20:35:58.697450 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.697423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:58.697600 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.697461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:35:58.697600 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:58.697578    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:02.697563687 +0000 UTC m=+60.031904281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : configmap references non-existent config key: service-ca.crt
Jul 06 20:35:58.697600 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:58.697581    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:35:58.697735 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:58.697639    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:02.697621718 +0000 UTC m=+60.031962295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : secret "router-metrics-certs-default" not found
Jul 06 20:35:58.798840 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.798813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dj9\" (UniqueName: \"kubernetes.io/projected/d0176d10-25ff-4047-bc6c-746bc14f5141-kube-api-access-26dj9\") pod \"network-check-source-989f8b8dc-ktssk\" (UID: \"d0176d10-25ff-4047-bc6c-746bc14f5141\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"
Jul 06 20:35:58.866468 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.866433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:58.899500 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.899428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dj9\" (UniqueName: \"kubernetes.io/projected/d0176d10-25ff-4047-bc6c-746bc14f5141-kube-api-access-26dj9\") pod \"network-check-source-989f8b8dc-ktssk\" (UID: \"d0176d10-25ff-4047-bc6c-746bc14f5141\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"
Jul 06 20:35:58.910086 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.910064    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dj9\" (UniqueName: \"kubernetes.io/projected/d0176d10-25ff-4047-bc6c-746bc14f5141-kube-api-access-26dj9\") pod \"network-check-source-989f8b8dc-ktssk\" (UID: \"d0176d10-25ff-4047-bc6c-746bc14f5141\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"
Jul 06 20:35:58.938197 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.938170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:58.978181 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:58.978153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"
Jul 06 20:35:59.066975 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.066934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:59.109243 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.109215    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"]
Jul 06 20:35:59.131397 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.131369    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"]
Jul 06 20:35:59.131541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.131409    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk"]
Jul 06 20:35:59.131541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.131510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"
Jul 06 20:35:59.134773 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.134743    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:59.134865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.134774    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 06 20:35:59.134865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.134774    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4ld7c\""
Jul 06 20:35:59.148856 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:59.148834    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0176d10_25ff_4047_bc6c_746bc14f5141.slice/crio-3ed023515e24c72dce7ddb8269ddc8264f38a17b87bdd512c8cec97ec883a114 WatchSource:0}: Error finding container 3ed023515e24c72dce7ddb8269ddc8264f38a17b87bdd512c8cec97ec883a114: Status 404 returned error can't find the container with id 3ed023515e24c72dce7ddb8269ddc8264f38a17b87bdd512c8cec97ec883a114
Jul 06 20:35:59.172835 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.172809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:59.213214 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.213187    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fl8wb"]
Jul 06 20:35:59.230481 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.230460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.234886 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.234866    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 20:35:59.235008 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.234947    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 20:35:59.235008 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.234947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 20:35:59.235207 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.235020    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 20:35:59.235339 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.235323    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42v4d\""
Jul 06 20:35:59.251363 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.251340    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fl8wb"]
Jul 06 20:35:59.302150 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.302126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtq6p\" (UniqueName: \"kubernetes.io/projected/61dd7738-53ef-4ece-a8dd-ca81fa6557df-kube-api-access-wtq6p\") pod \"migrator-d57d58cc9-97pd6\" (UID: \"61dd7738-53ef-4ece-a8dd-ca81fa6557df\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"
Jul 06 20:35:59.312892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.312866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:35:59.402953 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.402863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-crio-socket\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.402953 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.402908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.403182 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.403070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-data-volume\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.403182 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.403105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.403263 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.403192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtq6p\" (UniqueName: \"kubernetes.io/projected/61dd7738-53ef-4ece-a8dd-ca81fa6557df-kube-api-access-wtq6p\") pod \"migrator-d57d58cc9-97pd6\" (UID: \"61dd7738-53ef-4ece-a8dd-ca81fa6557df\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"
Jul 06 20:35:59.403263 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.403219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lv7w\" (UniqueName: \"kubernetes.io/projected/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-api-access-7lv7w\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.413768 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.413745    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtq6p\" (UniqueName: \"kubernetes.io/projected/61dd7738-53ef-4ece-a8dd-ca81fa6557df-kube-api-access-wtq6p\") pod \"migrator-d57d58cc9-97pd6\" (UID: \"61dd7738-53ef-4ece-a8dd-ca81fa6557df\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"
Jul 06 20:35:59.440251 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.440226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"
Jul 06 20:35:59.503828 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.503802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-crio-socket\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.503937 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.503854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.504006 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.503949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-data-volume\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.504006 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.503992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.504129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.504039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv7w\" (UniqueName: \"kubernetes.io/projected/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-api-access-7lv7w\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.504287 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:59.504262    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:59.504593 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:59.504349    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:00.004324949 +0000 UTC m=+57.338665521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:35:59.504593 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.504436    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-data-volume\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.504593 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:59.504442    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:35:59.504593 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.504059    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-crio-socket\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.504593 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:59.504514    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:00.004498678 +0000 UTC m=+57.338839266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : configmap "kube-rbac-proxy" not found
Jul 06 20:35:59.516280 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.516251    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lv7w\" (UniqueName: \"kubernetes.io/projected/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-api-access-7lv7w\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:35:59.575493 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.575472    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6"]
Jul 06 20:35:59.577994 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:35:59.577951    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61dd7738_53ef_4ece_a8dd_ca81fa6557df.slice/crio-3c3d4682c9c924083463a1edd2101bafea2435e39432548c156e37f2741cbc8f WatchSource:0}: Error finding container 3c3d4682c9c924083463a1edd2101bafea2435e39432548c156e37f2741cbc8f: Status 404 returned error can't find the container with id 3c3d4682c9c924083463a1edd2101bafea2435e39432548c156e37f2741cbc8f
Jul 06 20:35:59.629777 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.629752    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:35:59.630151 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.630136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/0.log"
Jul 06 20:35:59.630194 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.630172    2582 generic.go:356] "Generic (PLEG): container finished" podID="fd23fc80-e745-4494-99c4-4dde8a5772f0" containerID="72b97fee70ac820c81fb4b67d1d138144ef57e90e47d9960feb602bc39eb06a9" exitCode=255
Jul 06 20:35:59.630286 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.630264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" event={"ID":"fd23fc80-e745-4494-99c4-4dde8a5772f0","Type":"ContainerDied","Data":"72b97fee70ac820c81fb4b67d1d138144ef57e90e47d9960feb602bc39eb06a9"}
Jul 06 20:35:59.630390 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.630359    2582 scope.go:117] "RemoveContainer" containerID="8a07832cda89a4314a908b88b1afc9e03de11f758c18f05cee5444a9612e3d65"
Jul 06 20:35:59.630583 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.630567    2582 scope.go:117] "RemoveContainer" containerID="72b97fee70ac820c81fb4b67d1d138144ef57e90e47d9960feb602bc39eb06a9"
Jul 06 20:35:59.630761 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:35:59.630740    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-pn8bl_openshift-console-operator(fd23fc80-e745-4494-99c4-4dde8a5772f0)\"" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" podUID="fd23fc80-e745-4494-99c4-4dde8a5772f0"
Jul 06 20:35:59.631483 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.631460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6" event={"ID":"61dd7738-53ef-4ece-a8dd-ca81fa6557df","Type":"ContainerStarted","Data":"3c3d4682c9c924083463a1edd2101bafea2435e39432548c156e37f2741cbc8f"}
Jul 06 20:35:59.634725 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:35:59.634702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk" event={"ID":"d0176d10-25ff-4047-bc6c-746bc14f5141","Type":"ContainerStarted","Data":"3ed023515e24c72dce7ddb8269ddc8264f38a17b87bdd512c8cec97ec883a114"}
Jul 06 20:36:00.007840 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:00.007798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:00.008084 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:00.007913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:00.008084 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:00.007931    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:00.008084 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:00.008016    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.007999952 +0000 UTC m=+58.342340524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:00.008084 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:00.008061    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:00.008227 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:00.008125    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.008110292 +0000 UTC m=+58.342450866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:00.523837 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:00.523775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:00.623142 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:00.623104    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:00.639146 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:00.639119    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:36:00.639614 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:00.639512    2582 scope.go:117] "RemoveContainer" containerID="72b97fee70ac820c81fb4b67d1d138144ef57e90e47d9960feb602bc39eb06a9"
Jul 06 20:36:00.639695 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:00.639679    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-pn8bl_openshift-console-operator(fd23fc80-e745-4494-99c4-4dde8a5772f0)\"" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" podUID="fd23fc80-e745-4494-99c4-4dde8a5772f0"
Jul 06 20:36:01.018082 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.018044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:01.018280 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.018127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:01.018280 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:01.018229    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:01.018476 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:01.018294    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:03.018273983 +0000 UTC m=+60.352614569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:01.018476 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:01.018230    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:01.018476 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:01.018359    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:03.018346258 +0000 UTC m=+60.352686831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:01.209350 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.209307    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:01.316841 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.316761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:01.632701 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.632584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:01.643508 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.643415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6" event={"ID":"61dd7738-53ef-4ece-a8dd-ca81fa6557df","Type":"ContainerStarted","Data":"f1f808595c10bb207e2d273aaf8c67ce7cbbaef5809f7e6005f93047938539ee"}
Jul 06 20:36:01.643508 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.643477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6" event={"ID":"61dd7738-53ef-4ece-a8dd-ca81fa6557df","Type":"ContainerStarted","Data":"1d80b51549c6987048f28b37e3378c894bbf20a53cf512503cc5f6197ef931cc"}
Jul 06 20:36:01.666726 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.666678    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-97pd6" podStartSLOduration=1.185932046 podStartE2EDuration="2.666662561s" totalImagesPullingTime="1.480730515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:01.666077943 +0000 UTC m=+59.000418557" watchObservedRunningTime="2026-07-06 20:36:01.666662561 +0000 UTC m=+59.001003155"
Jul 06 20:36:01.733258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.733203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:01.835097 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.835053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:01.935641 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:01.935606    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:02.193111 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:02.193025    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hx4xq_09d366b1-0bfa-49f1-8cfe-d68f9fd5dece/dns-node-resolver/0.log"
Jul 06 20:36:02.712449 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:02.712418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:02.734571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:02.734536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:02.734730 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:02.734582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:02.734730 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:02.734702    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:10.734689621 +0000 UTC m=+68.069030195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : configmap references non-existent config key: service-ca.crt
Jul 06 20:36:02.734730 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:02.734703    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:36:02.734856 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:02.734766    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:10.734749628 +0000 UTC m=+68.069090203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : secret "router-metrics-certs-default" not found
Jul 06 20:36:02.792385 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:02.792357    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4jcp4_700e472f-98d4-4970-b9ea-3b1d6ed575e6/node-ca/0.log"
Jul 06 20:36:02.813994 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:02.813951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:03.037816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:03.037778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:03.038021 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:03.037932    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:03.038021 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:03.037939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:03.038021 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:03.038002    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:07.037987598 +0000 UTC m=+64.372328171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:03.038159 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:03.038039    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:03.038159 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:03.038097    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:07.038081579 +0000 UTC m=+64.372422158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:03.528249 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:03.528216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:03.595419 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:03.595384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:03.649767 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:03.649734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk" event={"ID":"d0176d10-25ff-4047-bc6c-746bc14f5141","Type":"ContainerStarted","Data":"0d50a9f1976ca3454d90c352b8bb12f1a054c02b7b304cede82a5690ad3b7691"}
Jul 06 20:36:03.675314 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:03.675270    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ktssk" podStartSLOduration=2.207051084 podStartE2EDuration="5.675254466s" totalImagesPullingTime="3.468203382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:03.673138094 +0000 UTC m=+61.007478689" watchObservedRunningTime="2026-07-06 20:36:03.675254466 +0000 UTC m=+61.009595061"
Jul 06 20:36:03.993036 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:03.993012    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vr2td_a3c1ec3f-6328-48cc-bc69-d20eaab9e41c/serve-healthcheck-canary/0.log"
Jul 06 20:36:04.195898 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:04.195870    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-97pd6_61dd7738-53ef-4ece-a8dd-ca81fa6557df/migrator/0.log"
Jul 06 20:36:04.397791 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:04.397707    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-97pd6_61dd7738-53ef-4ece-a8dd-ca81fa6557df/graceful-termination/0.log"
Jul 06 20:36:04.517015 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:04.516984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:05.209126 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:05.209101    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2hcz"
Jul 06 20:36:05.361623 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:05.361594    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:36:05.361782 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:05.361678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:36:05.361907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:05.361896    2582 scope.go:117] "RemoveContainer" containerID="72b97fee70ac820c81fb4b67d1d138144ef57e90e47d9960feb602bc39eb06a9"
Jul 06 20:36:05.362058 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:05.362039    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-pn8bl_openshift-console-operator(fd23fc80-e745-4494-99c4-4dde8a5772f0)\"" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" podUID="fd23fc80-e745-4494-99c4-4dde8a5772f0"
Jul 06 20:36:05.656197 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:05.656124    2582 scope.go:117] "RemoveContainer" containerID="72b97fee70ac820c81fb4b67d1d138144ef57e90e47d9960feb602bc39eb06a9"
Jul 06 20:36:05.656344 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:05.656277    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-pn8bl_openshift-console-operator(fd23fc80-e745-4494-99c4-4dde8a5772f0)\"" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" podUID="fd23fc80-e745-4494-99c4-4dde8a5772f0"
Jul 06 20:36:07.069333 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:07.069292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:07.069715 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:07.069358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:07.069715 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:07.069441    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:36:07.069715 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:07.069442    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:07.069715 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:07.069493    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:15.069480558 +0000 UTC m=+72.403821131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : configmap "kube-rbac-proxy" not found
Jul 06 20:36:07.069715 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:07.069507    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:15.069500689 +0000 UTC m=+72.403841262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:08.881177 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:08.881141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:36:08.881540 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:08.881216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:36:08.884167 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:08.884152    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:36:08.884403 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:08.884387    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:36:08.891718 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:08.891698    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 20:36:08.891761 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:08.891752    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs podName:c0ec3ed5-d7c9-4842-a336-9579edc57bf4 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:12.891737994 +0000 UTC m=+130.226078567 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs") pod "network-metrics-daemon-j8bqp" (UID: "c0ec3ed5-d7c9-4842-a336-9579edc57bf4") : secret "metrics-daemon-secret" not found
Jul 06 20:36:08.893856 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:08.893834    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/042b84ca-928d-462f-8a34-fb128944ed65-original-pull-secret\") pod \"global-pull-secret-syncer-w86tz\" (UID: \"042b84ca-928d-462f-8a34-fb128944ed65\") " pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:36:09.049054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.049015    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w86tz"
Jul 06 20:36:09.082180 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.082151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:36:09.084651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.084623    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88bph\" (UniqueName: \"kubernetes.io/projected/78929ea1-2b61-4d14-9dba-1890eeb9e9b2-kube-api-access-88bph\") pod \"network-check-target-p5hbx\" (UID: \"78929ea1-2b61-4d14-9dba-1890eeb9e9b2\") " pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:36:09.185049 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.185010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w86tz"]
Jul 06 20:36:09.189432 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:09.189396    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042b84ca_928d_462f_8a34_fb128944ed65.slice/crio-d570ed49fb5c2492cd4f91ab570e6ac14ee6e8b6de93af793b232b9a5d736cff WatchSource:0}: Error finding container d570ed49fb5c2492cd4f91ab570e6ac14ee6e8b6de93af793b232b9a5d736cff: Status 404 returned error can't find the container with id d570ed49fb5c2492cd4f91ab570e6ac14ee6e8b6de93af793b232b9a5d736cff
Jul 06 20:36:09.358400 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.358372    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6bf8b\""
Jul 06 20:36:09.366123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.366104    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:36:09.450183 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.450158    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:09.492332 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.492294    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p5hbx"]
Jul 06 20:36:09.496157 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:09.496131    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78929ea1_2b61_4d14_9dba_1890eeb9e9b2.slice/crio-ecc0beaa8f5f987c40cfbbcad74fceec0f5ef59418c1c433766b06a783e3fd1d WatchSource:0}: Error finding container ecc0beaa8f5f987c40cfbbcad74fceec0f5ef59418c1c433766b06a783e3fd1d: Status 404 returned error can't find the container with id ecc0beaa8f5f987c40cfbbcad74fceec0f5ef59418c1c433766b06a783e3fd1d
Jul 06 20:36:09.644003 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.643953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:09.668671 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.668640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p5hbx" event={"ID":"78929ea1-2b61-4d14-9dba-1890eeb9e9b2","Type":"ContainerStarted","Data":"301f330b8392beccd318a631206341d52f62e5ccb807f0797e60b5cf7f1833ed"}
Jul 06 20:36:09.668810 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.668684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p5hbx" event={"ID":"78929ea1-2b61-4d14-9dba-1890eeb9e9b2","Type":"ContainerStarted","Data":"ecc0beaa8f5f987c40cfbbcad74fceec0f5ef59418c1c433766b06a783e3fd1d"}
Jul 06 20:36:09.668849 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.668836    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:36:09.669724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.669698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w86tz" event={"ID":"042b84ca-928d-462f-8a34-fb128944ed65","Type":"ContainerStarted","Data":"d570ed49fb5c2492cd4f91ab570e6ac14ee6e8b6de93af793b232b9a5d736cff"}
Jul 06 20:36:09.687799 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.687753    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p5hbx" podStartSLOduration=66.687739552 podStartE2EDuration="1m6.687739552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:09.687524884 +0000 UTC m=+67.021865491" watchObservedRunningTime="2026-07-06 20:36:09.687739552 +0000 UTC m=+67.022080147"
Jul 06 20:36:09.793570 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.793495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:09.890040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.890008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:36:09.890417 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:09.890162    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:09.890417 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:09.890226    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls podName:05d6c585-6fc0-45f4-a258-e59b0d234bf9 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:41.890210301 +0000 UTC m=+99.224550887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls") pod "dns-default-rps8j" (UID: "05d6c585-6fc0-45f4-a258-e59b0d234bf9") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:09.894596 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:09.894572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:10.002104 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:10.002058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:10.108813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:10.108725    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:10.311761 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:10.311729    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:10.797019 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:10.796979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:10.797218 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:10.797048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:10.797218 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:10.797136    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 20:36:10.797218 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:10.797211    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:26.797190812 +0000 UTC m=+84.131531387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : secret "router-metrics-certs-default" not found
Jul 06 20:36:10.797359 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:10.797228    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle podName:30cb12a1-e55d-432b-b856-258d10383502 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:26.79722112 +0000 UTC m=+84.131561694 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle") pod "router-default-757d9b65f5-7xf7w" (UID: "30cb12a1-e55d-432b-b856-258d10383502") : configmap references non-existent config key: service-ca.crt
Jul 06 20:36:11.488710 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:11.488673    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:11.699576 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:11.699543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:11.889778 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:11.889700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:12.087057 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:12.087025    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:12.203260 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:12.203224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:12.305567 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:12.305531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:13.092174 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:13.092138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:13.285308 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:13.285278    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:13.518797 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:13.518763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:13.691651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:13.691623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:13.890593 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:13.890561    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.294569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:14.294534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.353667 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:14.353642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.490093 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:14.490064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:14.686777 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:14.686749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w86tz" event={"ID":"042b84ca-928d-462f-8a34-fb128944ed65","Type":"ContainerStarted","Data":"95b260b2285900d52f9d7ea8be168220cc80af4485799923e9fd82eb41d5f329"}
Jul 06 20:36:14.706294 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:14.706246    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w86tz" podStartSLOduration=67.244921458 podStartE2EDuration="1m11.706230626s" totalImagesPullingTime="4.461309168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:14.70499125 +0000 UTC m=+72.039331845" watchObservedRunningTime="2026-07-06 20:36:14.706230626 +0000 UTC m=+72.040571267"
Jul 06 20:36:14.889091 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:14.889061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:15.134233 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:15.134146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:15.134233 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:15.134216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:15.134414 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:15.134319    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:15.134414 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:15.134376    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls podName:d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.134361689 +0000 UTC m=+88.468702266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fl8wb" (UID: "d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:36:15.134643 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:15.134626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:15.693664 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:15.693637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:15.810364 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:15.810327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:15.913349 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:15.913312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:16.089091 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:16.089015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:16.152186 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:16.152161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:16.497352 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:16.497325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.300723 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.300690    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.331043 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.331017    2582 scope.go:117] "RemoveContainer" containerID="72b97fee70ac820c81fb4b67d1d138144ef57e90e47d9960feb602bc39eb06a9"
Jul 06 20:36:17.495558 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.495527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:17.697270 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.697244    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:36:17.697426 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.697311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" event={"ID":"fd23fc80-e745-4494-99c4-4dde8a5772f0","Type":"ContainerStarted","Data":"dee5601d31624bee236e9af036da3fc9de6af6ced2f77203e861876142b0b860"}
Jul 06 20:36:17.697649 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.697623    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:36:17.725734 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.725676    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl" podStartSLOduration=20.115861139 podStartE2EDuration="22.725656106s" totalImagesPullingTime="2.609794967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:17.725004969 +0000 UTC m=+75.059345566" watchObservedRunningTime="2026-07-06 20:36:17.725656106 +0000 UTC m=+75.059996702"
Jul 06 20:36:17.762283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.762258    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-pn8bl"
Jul 06 20:36:17.900902 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:17.900869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.046201 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:18.046121    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.141772 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:18.141738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.242532 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:18.242500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.627921 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:18.627888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:18.728425 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:18.728398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.051731 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:19.051701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.150853 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:19.150827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.285736 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:19.285709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.356883 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:19.356801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.460718 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:19.460687    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.561852 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:19.561827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:19.662295 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:19.662213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.305123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:20.305080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.507948 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:20.507917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.703470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:20.703441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:20.877479 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:20.877445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:26.829144 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:26.828995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:26.829144 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:26.829055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:26.829773 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:26.829751    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30cb12a1-e55d-432b-b856-258d10383502-service-ca-bundle\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:26.831469 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:26.831450    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cb12a1-e55d-432b-b856-258d10383502-metrics-certs\") pod \"router-default-757d9b65f5-7xf7w\" (UID: \"30cb12a1-e55d-432b-b856-258d10383502\") " pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:26.948646 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:26.948601    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:27.049728 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.049695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:27.068802 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.068781    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8tnrg\""
Jul 06 20:36:27.077040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.077022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:27.201476 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.201444    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-757d9b65f5-7xf7w"]
Jul 06 20:36:27.205793 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:27.205762    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cb12a1_e55d_432b_b856_258d10383502.slice/crio-0c00976e366b8e4fbde5b6939aebabf614bc50f6994a379526d611eba19df1a2 WatchSource:0}: Error finding container 0c00976e366b8e4fbde5b6939aebabf614bc50f6994a379526d611eba19df1a2: Status 404 returned error can't find the container with id 0c00976e366b8e4fbde5b6939aebabf614bc50f6994a379526d611eba19df1a2
Jul 06 20:36:27.729153 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.729114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-757d9b65f5-7xf7w" event={"ID":"30cb12a1-e55d-432b-b856-258d10383502","Type":"ContainerStarted","Data":"8e11116b4a456327165e295d6a93ed8713d64caea067c97a9d64a18d992589a0"}
Jul 06 20:36:27.729322 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.729174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-757d9b65f5-7xf7w" event={"ID":"30cb12a1-e55d-432b-b856-258d10383502","Type":"ContainerStarted","Data":"0c00976e366b8e4fbde5b6939aebabf614bc50f6994a379526d611eba19df1a2"}
Jul 06 20:36:27.755474 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.755428    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-757d9b65f5-7xf7w" podStartSLOduration=33.755411773 podStartE2EDuration="33.755411773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:27.75446327 +0000 UTC m=+85.088803865" watchObservedRunningTime="2026-07-06 20:36:27.755411773 +0000 UTC m=+85.089752367"
Jul 06 20:36:27.955876 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.955846    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"]
Jul 06 20:36:27.957787 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.957772    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:27.961566 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.961545    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 20:36:27.961698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.961588    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 20:36:27.961877 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.961861    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 20:36:27.962017 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.962002    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 06 20:36:27.962131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.962116    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hjbx4\""
Jul 06 20:36:27.974432 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:27.974403    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"]
Jul 06 20:36:28.037222 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.037126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f46dbb70-61d8-4e9a-9868-b5d24232cede-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5576b67cf4-nlj9l\" (UID: \"f46dbb70-61d8-4e9a-9868-b5d24232cede\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:28.037222 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.037169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qxf\" (UniqueName: \"kubernetes.io/projected/f46dbb70-61d8-4e9a-9868-b5d24232cede-kube-api-access-z9qxf\") pod \"managed-serviceaccount-addon-agent-5576b67cf4-nlj9l\" (UID: \"f46dbb70-61d8-4e9a-9868-b5d24232cede\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:28.077197 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.077164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:28.080018 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.079949    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:28.138208 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.138170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f46dbb70-61d8-4e9a-9868-b5d24232cede-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5576b67cf4-nlj9l\" (UID: \"f46dbb70-61d8-4e9a-9868-b5d24232cede\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:28.138208 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.138203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qxf\" (UniqueName: \"kubernetes.io/projected/f46dbb70-61d8-4e9a-9868-b5d24232cede-kube-api-access-z9qxf\") pod \"managed-serviceaccount-addon-agent-5576b67cf4-nlj9l\" (UID: \"f46dbb70-61d8-4e9a-9868-b5d24232cede\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:28.140694 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.140671    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f46dbb70-61d8-4e9a-9868-b5d24232cede-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5576b67cf4-nlj9l\" (UID: \"f46dbb70-61d8-4e9a-9868-b5d24232cede\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:28.161166 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.161112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.164466 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.164436    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qxf\" (UniqueName: \"kubernetes.io/projected/f46dbb70-61d8-4e9a-9868-b5d24232cede-kube-api-access-z9qxf\") pod \"managed-serviceaccount-addon-agent-5576b67cf4-nlj9l\" (UID: \"f46dbb70-61d8-4e9a-9868-b5d24232cede\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:28.171531 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.171507    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55cdb97ff5-g8zn2"]
Jul 06 20:36:28.173369 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.173353    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.176160 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.176131    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 20:36:28.176266 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.176209    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 20:36:28.176423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.176410    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nc576\""
Jul 06 20:36:28.177093 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.177079    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 20:36:28.185405 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.185364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 20:36:28.191936 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.191912    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55cdb97ff5-g8zn2"]
Jul 06 20:36:28.238900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.238871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73657f86-99d3-4f35-8bd4-a9fc7b56c977-ca-trust-extracted\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.239088 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.238910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-registry-tls\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.239088 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.238930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-bound-sa-token\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.239088 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.238949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73657f86-99d3-4f35-8bd4-a9fc7b56c977-registry-certificates\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.239088 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.239058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73657f86-99d3-4f35-8bd4-a9fc7b56c977-trusted-ca\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.239298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.239102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rbp\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-kube-api-access-k9rbp\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.239298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.239205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73657f86-99d3-4f35-8bd4-a9fc7b56c977-installation-pull-secrets\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.239298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.239257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73657f86-99d3-4f35-8bd4-a9fc7b56c977-image-registry-private-configuration\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.256690 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.256662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.280073 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.280043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"
Jul 06 20:36:28.339592 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-registry-tls\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.339741 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-bound-sa-token\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.339741 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73657f86-99d3-4f35-8bd4-a9fc7b56c977-registry-certificates\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.339741 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73657f86-99d3-4f35-8bd4-a9fc7b56c977-trusted-ca\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.339741 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rbp\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-kube-api-access-k9rbp\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.339933 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73657f86-99d3-4f35-8bd4-a9fc7b56c977-installation-pull-secrets\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.339933 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73657f86-99d3-4f35-8bd4-a9fc7b56c977-image-registry-private-configuration\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.339933 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.339831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73657f86-99d3-4f35-8bd4-a9fc7b56c977-ca-trust-extracted\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.340744 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.340717    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73657f86-99d3-4f35-8bd4-a9fc7b56c977-registry-certificates\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.340744 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.340713    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73657f86-99d3-4f35-8bd4-a9fc7b56c977-ca-trust-extracted\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.341089 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.341063    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73657f86-99d3-4f35-8bd4-a9fc7b56c977-trusted-ca\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.342181 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.342141    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-registry-tls\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.343116 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.343094    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73657f86-99d3-4f35-8bd4-a9fc7b56c977-image-registry-private-configuration\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.343303 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.343282    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73657f86-99d3-4f35-8bd4-a9fc7b56c977-installation-pull-secrets\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.353461 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.353435    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-bound-sa-token\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.354307 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.354284    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rbp\" (UniqueName: \"kubernetes.io/projected/73657f86-99d3-4f35-8bd4-a9fc7b56c977-kube-api-access-k9rbp\") pod \"image-registry-55cdb97ff5-g8zn2\" (UID: \"73657f86-99d3-4f35-8bd4-a9fc7b56c977\") " pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.364368 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.364341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.417225 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.417201    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l"]
Jul 06 20:36:28.419149 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:28.419122    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf46dbb70_61d8_4e9a_9868_b5d24232cede.slice/crio-3cecb6a982220878ad2501c3d2810ad60ed30a6fc955e8b5b9731feafd87c412 WatchSource:0}: Error finding container 3cecb6a982220878ad2501c3d2810ad60ed30a6fc955e8b5b9731feafd87c412: Status 404 returned error can't find the container with id 3cecb6a982220878ad2501c3d2810ad60ed30a6fc955e8b5b9731feafd87c412
Jul 06 20:36:28.466382 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.466345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.486392 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.486353    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:28.564787 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.564760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:28.617588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.617565    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55cdb97ff5-g8zn2"]
Jul 06 20:36:28.619322 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:28.619294    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73657f86_99d3_4f35_8bd4_a9fc7b56c977.slice/crio-7cb4983aee59f4d3339f7581bab384f1f579e953ef6d93a3b2a0a4fc25e9e6fe WatchSource:0}: Error finding container 7cb4983aee59f4d3339f7581bab384f1f579e953ef6d93a3b2a0a4fc25e9e6fe: Status 404 returned error can't find the container with id 7cb4983aee59f4d3339f7581bab384f1f579e953ef6d93a3b2a0a4fc25e9e6fe
Jul 06 20:36:28.733726 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.733693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l" event={"ID":"f46dbb70-61d8-4e9a-9868-b5d24232cede","Type":"ContainerStarted","Data":"3cecb6a982220878ad2501c3d2810ad60ed30a6fc955e8b5b9731feafd87c412"}
Jul 06 20:36:28.735186 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.735160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2" event={"ID":"73657f86-99d3-4f35-8bd4-a9fc7b56c977","Type":"ContainerStarted","Data":"9a2d9617fd861c701499e325d6e167201dd8e015af88fa8fe482c2037d1a6e86"}
Jul 06 20:36:28.735326 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.735199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2" event={"ID":"73657f86-99d3-4f35-8bd4-a9fc7b56c977","Type":"ContainerStarted","Data":"7cb4983aee59f4d3339f7581bab384f1f579e953ef6d93a3b2a0a4fc25e9e6fe"}
Jul 06 20:36:28.735508 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.735487    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:28.736739 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.736722    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-757d9b65f5-7xf7w"
Jul 06 20:36:28.764225 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:28.764171    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2" podStartSLOduration=0.764154953 podStartE2EDuration="764.154953ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:28.762742736 +0000 UTC m=+86.097083328" watchObservedRunningTime="2026-07-06 20:36:28.764154953 +0000 UTC m=+86.098495548"
Jul 06 20:36:29.167332 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:29.167258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.290471 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:29.290383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.390587 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:29.390551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.485597 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:29.485563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:29.739449 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:29.739347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:31.165145 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.165026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:31.167423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.167398    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fl8wb\" (UID: \"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8\") " pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:31.341882 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.341845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42v4d\""
Jul 06 20:36:31.349393 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.349370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fl8wb"
Jul 06 20:36:31.485243 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.485209    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fl8wb"]
Jul 06 20:36:31.497588 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:31.497558    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02ba9b3_20a9_4423_82ae_2c6fb40c4cd8.slice/crio-e641650f1cf988ea62d6e3305c1a146d5dfd59d9e85bbed7140fb32f7c120f61 WatchSource:0}: Error finding container e641650f1cf988ea62d6e3305c1a146d5dfd59d9e85bbed7140fb32f7c120f61: Status 404 returned error can't find the container with id e641650f1cf988ea62d6e3305c1a146d5dfd59d9e85bbed7140fb32f7c120f61
Jul 06 20:36:31.533327 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.533300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:31.749514 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.749412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l" event={"ID":"f46dbb70-61d8-4e9a-9868-b5d24232cede","Type":"ContainerStarted","Data":"4cd0778ef7d1451470076118ae21242cdb1a9c43cad9d2ca30885690f3522962"}
Jul 06 20:36:31.750572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.750550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fl8wb" event={"ID":"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8","Type":"ContainerStarted","Data":"e641650f1cf988ea62d6e3305c1a146d5dfd59d9e85bbed7140fb32f7c120f61"}
Jul 06 20:36:31.767053 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:31.766917    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5576b67cf4-nlj9l" podStartSLOduration=2.486270953 podStartE2EDuration="4.766900126s" totalImagesPullingTime="2.280629173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:31.7665398 +0000 UTC m=+89.100880398" watchObservedRunningTime="2026-07-06 20:36:31.766900126 +0000 UTC m=+89.101240723"
Jul 06 20:36:33.387176 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.387128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:33.474116 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.474071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:33.478415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.478388    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff9c8fd7b-kkh6q"]
Jul 06 20:36:33.491687 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.491657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.492886 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.492861    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff9c8fd7b-kkh6q"]
Jul 06 20:36:33.495032 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.494726    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 06 20:36:33.495032 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.494766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 06 20:36:33.495032 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.494726    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n2dlm\""
Jul 06 20:36:33.496375 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.496152    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 06 20:36:33.496375 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.496171    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 06 20:36:33.496375 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.496190    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 06 20:36:33.496605 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.496425    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 06 20:36:33.496605 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.496505    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 06 20:36:33.500849 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.500824    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 06 20:36:33.552715 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.552682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:33.586208 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.586176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-oauth-config\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.586367 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.586214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-config\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.586367 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.586247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-service-ca\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.586367 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.586270    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs87d\" (UniqueName: \"kubernetes.io/projected/16b1624a-3cac-49a2-b63f-acbe45d9071c-kube-api-access-xs87d\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.586367 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.586337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-oauth-serving-cert\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.586563 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.586426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-serving-cert\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.586563 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.586453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-trusted-ca-bundle\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.622659 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.622620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:33.687016 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.686957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-oauth-config\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.687016 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.687021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-config\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.687283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.687065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-service-ca\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.687283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.687096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs87d\" (UniqueName: \"kubernetes.io/projected/16b1624a-3cac-49a2-b63f-acbe45d9071c-kube-api-access-xs87d\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.687283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.687159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-oauth-serving-cert\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.687283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.687227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-serving-cert\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.687283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.687245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-trusted-ca-bundle\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.688184 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.688146    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-trusted-ca-bundle\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.688311 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.688263    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-service-ca\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.688522 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.688499    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-oauth-serving-cert\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.688844 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.688826    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-config\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.689855 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.689826    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-oauth-config\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.690472 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.690450    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-serving-cert\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.697003 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.696950    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs87d\" (UniqueName: \"kubernetes.io/projected/16b1624a-3cac-49a2-b63f-acbe45d9071c-kube-api-access-xs87d\") pod \"console-5ff9c8fd7b-kkh6q\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") " pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:33.804386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:33.804349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:35.189048 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:35.189010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff9c8fd7b-kkh6q"]
Jul 06 20:36:35.192656 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:35.192630    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b1624a_3cac_49a2_b63f_acbe45d9071c.slice/crio-0cfa6262f4352ccaed76f9f50fac55112fca03cf59a8cdc1b7e0d9bc33abe956 WatchSource:0}: Error finding container 0cfa6262f4352ccaed76f9f50fac55112fca03cf59a8cdc1b7e0d9bc33abe956: Status 404 returned error can't find the container with id 0cfa6262f4352ccaed76f9f50fac55112fca03cf59a8cdc1b7e0d9bc33abe956
Jul 06 20:36:35.768416 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:35.768376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fl8wb" event={"ID":"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8","Type":"ContainerStarted","Data":"c206eef053aba0eb62381bea9c313f88eeb15bea1ecce11fdf62917bccae50ec"}
Jul 06 20:36:35.769792 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:35.769760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9c8fd7b-kkh6q" event={"ID":"16b1624a-3cac-49a2-b63f-acbe45d9071c","Type":"ContainerStarted","Data":"0cfa6262f4352ccaed76f9f50fac55112fca03cf59a8cdc1b7e0d9bc33abe956"}
Jul 06 20:36:36.775534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:36.775496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fl8wb" event={"ID":"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8","Type":"ContainerStarted","Data":"fed6e6cf62ffdf4d310397528edfe47cdca59ae59961904ba984253d98273247"}
Jul 06 20:36:37.283572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:37.283541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:37.352301 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:37.352269    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:37.781981 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:37.781935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9c8fd7b-kkh6q" event={"ID":"16b1624a-3cac-49a2-b63f-acbe45d9071c","Type":"ContainerStarted","Data":"344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307"}
Jul 06 20:36:37.808338 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:37.808270    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff9c8fd7b-kkh6q" podStartSLOduration=2.285716239 podStartE2EDuration="4.808250168s" totalImagesPullingTime="2.522533929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:37.808084846 +0000 UTC m=+95.142425468" watchObservedRunningTime="2026-07-06 20:36:37.808250168 +0000 UTC m=+95.142590766"
Jul 06 20:36:40.479619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:40.479584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:40.675855 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:40.675828    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p5hbx"
Jul 06 20:36:40.793302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:40.793223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fl8wb" event={"ID":"d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8","Type":"ContainerStarted","Data":"30c7af890fa5ebfd27540b6f692ca209d29117376ea657b591be12150230fbdc"}
Jul 06 20:36:40.816661 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:40.816597    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fl8wb" podStartSLOduration=33.678892843 podStartE2EDuration="41.816576161s" totalImagesPullingTime="8.137683318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:40.815498586 +0000 UTC m=+98.149839181" watchObservedRunningTime="2026-07-06 20:36:40.816576161 +0000 UTC m=+98.150916763"
Jul 06 20:36:40.817531 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:40.817507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:41.958948 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:41.958911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:36:41.961230 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:41.961204    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d6c585-6fc0-45f4-a258-e59b0d234bf9-metrics-tls\") pod \"dns-default-rps8j\" (UID: \"05d6c585-6fc0-45f4-a258-e59b0d234bf9\") " pod="openshift-dns/dns-default-rps8j"
Jul 06 20:36:42.249608 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.249520    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dqcct\""
Jul 06 20:36:42.257252 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.257226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rps8j"
Jul 06 20:36:42.388341 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.388317    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rps8j"]
Jul 06 20:36:42.390640 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:42.390614    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d6c585_6fc0_45f4_a258_e59b0d234bf9.slice/crio-0e144947136a66f85ac7e1ab61daab40e590f1b2a1f3403478a783bbec3148e5 WatchSource:0}: Error finding container 0e144947136a66f85ac7e1ab61daab40e590f1b2a1f3403478a783bbec3148e5: Status 404 returned error can't find the container with id 0e144947136a66f85ac7e1ab61daab40e590f1b2a1f3403478a783bbec3148e5
Jul 06 20:36:42.556618 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.556529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:42.665316 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.665282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:42.801677 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.801633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rps8j" event={"ID":"05d6c585-6fc0-45f4-a258-e59b0d234bf9","Type":"ContainerStarted","Data":"0e144947136a66f85ac7e1ab61daab40e590f1b2a1f3403478a783bbec3148e5"}
Jul 06 20:36:42.806704 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.806630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:42.878845 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.878812    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9vj4l"]
Jul 06 20:36:42.881447 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.881425    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.884871 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.884847    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 20:36:42.886338 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.886318    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 20:36:42.886495 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.886367    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 20:36:42.886571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.886318    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 20:36:42.886571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.886326    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 20:36:42.886571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.886562    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 20:36:42.887079 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.887050    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gl26r\""
Jul 06 20:36:42.928397 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.928361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:42.967294 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-sys\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-tls\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxj7\" (UniqueName: \"kubernetes.io/projected/e02d2f7b-67e1-4405-8c76-a22f6247ab73-kube-api-access-vbxj7\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-wtmp\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967473    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e02d2f7b-67e1-4405-8c76-a22f6247ab73-metrics-client-ca\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-root\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:42.967700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:42.967582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-textfile\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.021132 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.021091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.069022 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.068937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-wtmp\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069022 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.068987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069022 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e02d2f7b-67e1-4405-8c76-a22f6247ab73-metrics-client-ca\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-root\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069132    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-wtmp\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-textfile\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-sys\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-tls\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069589 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxj7\" (UniqueName: \"kubernetes.io/projected/e02d2f7b-67e1-4405-8c76-a22f6247ab73-kube-api-access-vbxj7\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069589 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069520    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-textfile\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069587    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-root\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e02d2f7b-67e1-4405-8c76-a22f6247ab73-sys\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069661    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.069685 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:43.069677    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 20:36:43.069874 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:43.069753    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-tls podName:e02d2f7b-67e1-4405-8c76-a22f6247ab73 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:43.569735172 +0000 UTC m=+100.904075762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-tls") pod "node-exporter-9vj4l" (UID: "e02d2f7b-67e1-4405-8c76-a22f6247ab73") : secret "node-exporter-tls" not found
Jul 06 20:36:43.069924 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.069880    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e02d2f7b-67e1-4405-8c76-a22f6247ab73-metrics-client-ca\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.071956 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.071932    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.080878 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.080856    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxj7\" (UniqueName: \"kubernetes.io/projected/e02d2f7b-67e1-4405-8c76-a22f6247ab73-kube-api-access-vbxj7\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.128079 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.128030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.308402 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.308360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.409664 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.409550    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.574330 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.574287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-tls\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.576878 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.576851    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e02d2f7b-67e1-4405-8c76-a22f6247ab73-node-exporter-tls\") pod \"node-exporter-9vj4l\" (UID: \"e02d2f7b-67e1-4405-8c76-a22f6247ab73\") " pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.647695 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.647654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.783859 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.783821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.792834 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.792805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vj4l"
Jul 06 20:36:43.802318 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:43.802291    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02d2f7b_67e1_4405_8c76_a22f6247ab73.slice/crio-30e8615febd6e45ee09841ddebdaedaafd95f18ba4e8bdda8326b25c5534b5b7 WatchSource:0}: Error finding container 30e8615febd6e45ee09841ddebdaedaafd95f18ba4e8bdda8326b25c5534b5b7: Status 404 returned error can't find the container with id 30e8615febd6e45ee09841ddebdaedaafd95f18ba4e8bdda8326b25c5534b5b7
Jul 06 20:36:43.804812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.804784    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:43.804917 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.804848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:43.810033 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.810009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:43.884547 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.884511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:43.986542 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:43.986516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.084733 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.084653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.183285 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.183245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:44.810280 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.810240    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vj4l" event={"ID":"e02d2f7b-67e1-4405-8c76-a22f6247ab73","Type":"ContainerStarted","Data":"30e8615febd6e45ee09841ddebdaedaafd95f18ba4e8bdda8326b25c5534b5b7"}
Jul 06 20:36:44.812550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.812523    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rps8j" event={"ID":"05d6c585-6fc0-45f4-a258-e59b0d234bf9","Type":"ContainerStarted","Data":"344fd4ff197deb836c4157656f2ee106d34ae5c006478e02b4f882a15fd19827"}
Jul 06 20:36:44.812693 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.812579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rps8j" event={"ID":"05d6c585-6fc0-45f4-a258-e59b0d234bf9","Type":"ContainerStarted","Data":"164bc7994d0dfd111e14d4a9d581c72ce54f3c9e31d1faa8cc711a4ff11e5f84"}
Jul 06 20:36:44.812839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.812807    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rps8j"
Jul 06 20:36:44.817021 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.816999    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:36:44.833505 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.833450    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rps8j" podStartSLOduration=66.40441021 podStartE2EDuration="1m7.833428658s" totalImagesPullingTime="1.429018448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:44.832432356 +0000 UTC m=+102.166772988" watchObservedRunningTime="2026-07-06 20:36:44.833428658 +0000 UTC m=+102.167769254"
Jul 06 20:36:44.913191 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.913161    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-687687fc7d-drjss"]
Jul 06 20:36:44.921947 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.921924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:44.925013 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.924990    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 06 20:36:44.925197 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.925179    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 06 20:36:44.925337 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.925251    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 06 20:36:44.925337 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.925282    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-576mm\""
Jul 06 20:36:44.925337 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.925310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 06 20:36:44.925470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.925352    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1sr09bofpsja9\""
Jul 06 20:36:44.925606 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.925591    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 06 20:36:44.935736 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:44.935713    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-687687fc7d-drjss"]
Jul 06 20:36:45.093529 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-tls\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.093892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.093892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.093892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa0ab60a-9947-406c-9495-7f4cf0512c8b-metrics-client-ca\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.093892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt74g\" (UniqueName: \"kubernetes.io/projected/fa0ab60a-9947-406c-9495-7f4cf0512c8b-kube-api-access-wt74g\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.093892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-grpc-tls\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.093892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.093892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.093858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.194853 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.194817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-tls\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.195074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.194873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.195074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.194910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.195074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.194942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa0ab60a-9947-406c-9495-7f4cf0512c8b-metrics-client-ca\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.195074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.195009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt74g\" (UniqueName: \"kubernetes.io/projected/fa0ab60a-9947-406c-9495-7f4cf0512c8b-kube-api-access-wt74g\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.195074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.195052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-grpc-tls\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.195410 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.195144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.195410 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.195173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.196484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.196457    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa0ab60a-9947-406c-9495-7f4cf0512c8b-metrics-client-ca\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.198466 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.198425    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.198695 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.198596    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.198883 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.198851    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.198956 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.198862    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-tls\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.199250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.199202    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.200149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.200124    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa0ab60a-9947-406c-9495-7f4cf0512c8b-secret-grpc-tls\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.203810 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.203783    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt74g\" (UniqueName: \"kubernetes.io/projected/fa0ab60a-9947-406c-9495-7f4cf0512c8b-kube-api-access-wt74g\") pod \"thanos-querier-687687fc7d-drjss\" (UID: \"fa0ab60a-9947-406c-9495-7f4cf0512c8b\") " pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.231829 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.231807    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:45.364109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.364032    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-687687fc7d-drjss"]
Jul 06 20:36:45.367392 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:45.367367    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0ab60a_9947_406c_9495_7f4cf0512c8b.slice/crio-af14ec00edb57d3a3ba97f9e018f21c0871fcfc9e65aedaa861b59a80ade3fd1 WatchSource:0}: Error finding container af14ec00edb57d3a3ba97f9e018f21c0871fcfc9e65aedaa861b59a80ade3fd1: Status 404 returned error can't find the container with id af14ec00edb57d3a3ba97f9e018f21c0871fcfc9e65aedaa861b59a80ade3fd1
Jul 06 20:36:45.788980 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.788931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:45.817348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.817314    2582 generic.go:356] "Generic (PLEG): container finished" podID="e02d2f7b-67e1-4405-8c76-a22f6247ab73" containerID="933aa3426de70f4b07e66c55ad11a44fdfd9853c965b850b64475b73732b4db5" exitCode=0
Jul 06 20:36:45.817548 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.817400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vj4l" event={"ID":"e02d2f7b-67e1-4405-8c76-a22f6247ab73","Type":"ContainerDied","Data":"933aa3426de70f4b07e66c55ad11a44fdfd9853c965b850b64475b73732b4db5"}
Jul 06 20:36:45.818672 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.818649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" event={"ID":"fa0ab60a-9947-406c-9495-7f4cf0512c8b","Type":"ContainerStarted","Data":"af14ec00edb57d3a3ba97f9e018f21c0871fcfc9e65aedaa861b59a80ade3fd1"}
Jul 06 20:36:45.862550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:45.862524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:46.087159 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.087078    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:46.248163 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.248110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:46.342846 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.342763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:46.826027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.825990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vj4l" event={"ID":"e02d2f7b-67e1-4405-8c76-a22f6247ab73","Type":"ContainerStarted","Data":"d72f5b0e4c4521aa6439ef0149813cc947cf0a3ea5897f41a0c738c77a922a3e"}
Jul 06 20:36:46.826222 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.826042    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vj4l" event={"ID":"e02d2f7b-67e1-4405-8c76-a22f6247ab73","Type":"ContainerStarted","Data":"eaf1533545dd9c761213e865dac39566e5f070faacae1cb73f2068cd37cc8018"}
Jul 06 20:36:46.857689 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.857637    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9vj4l" podStartSLOduration=3.553712202 podStartE2EDuration="4.857621739s" totalImagesPullingTime="1.239863705s" totalInitContainerRuntime="64.045832ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:46.857256575 +0000 UTC m=+104.191597194" watchObservedRunningTime="2026-07-06 20:36:46.857621739 +0000 UTC m=+104.191962334"
Jul 06 20:36:46.905007 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.904954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:46.982403 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:46.982359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:47.381949 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.381909    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86bddcdcdf-lprv2"]
Jul 06 20:36:47.415394 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.415343    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86bddcdcdf-lprv2"]
Jul 06 20:36:47.415548 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.415484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.424845 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.424781    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 06 20:36:47.426415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.425027    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 20:36:47.426415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.425187    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2asgee1drt8fm\""
Jul 06 20:36:47.426415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.425582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f5mrc\""
Jul 06 20:36:47.426415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.426150    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 06 20:36:47.431034 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.430406    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 06 20:36:47.516333 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.516299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4162691b-9b53-4f69-b26e-c10d6e157587-audit-log\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.516491 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.516338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4162691b-9b53-4f69-b26e-c10d6e157587-metrics-server-audit-profiles\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.516491 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.516428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-secret-metrics-server-client-certs\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.516588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.516565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-secret-metrics-server-tls\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.516633 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.516605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-client-ca-bundle\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.516696 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.516676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4162691b-9b53-4f69-b26e-c10d6e157587-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.516752 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.516737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrt2\" (UniqueName: \"kubernetes.io/projected/4162691b-9b53-4f69-b26e-c10d6e157587-kube-api-access-5lrt2\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.576639 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.576607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:47.583546 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.583519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"]
Jul 06 20:36:47.596416 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.596395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:47.598749 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.598730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"]
Jul 06 20:36:47.599247 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.599232    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-znjmh\""
Jul 06 20:36:47.599400 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.599385    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 06 20:36:47.617299 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-secret-metrics-server-tls\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.617427 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-client-ca-bundle\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.617427 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4162691b-9b53-4f69-b26e-c10d6e157587-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.617427 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrt2\" (UniqueName: \"kubernetes.io/projected/4162691b-9b53-4f69-b26e-c10d6e157587-kube-api-access-5lrt2\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.617577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4162691b-9b53-4f69-b26e-c10d6e157587-audit-log\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.617577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4162691b-9b53-4f69-b26e-c10d6e157587-metrics-server-audit-profiles\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.617724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-secret-metrics-server-client-certs\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.617899 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.617881    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4162691b-9b53-4f69-b26e-c10d6e157587-audit-log\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.618317 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.618291    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4162691b-9b53-4f69-b26e-c10d6e157587-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.618435 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.618414    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4162691b-9b53-4f69-b26e-c10d6e157587-metrics-server-audit-profiles\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.619937 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.619917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-secret-metrics-server-tls\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.620040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.619926    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-client-ca-bundle\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.620040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.619957    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4162691b-9b53-4f69-b26e-c10d6e157587-secret-metrics-server-client-certs\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.625790 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.625763    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrt2\" (UniqueName: \"kubernetes.io/projected/4162691b-9b53-4f69-b26e-c10d6e157587-kube-api-access-5lrt2\") pod \"metrics-server-86bddcdcdf-lprv2\" (UID: \"4162691b-9b53-4f69-b26e-c10d6e157587\") " pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.652682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.652620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:47.718892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.718860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a033849b-0d61-42a5-9aa8-34f6b749b22f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wzr47\" (UID: \"a033849b-0d61-42a5-9aa8-34f6b749b22f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:47.718892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.718898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnhq\" (UniqueName: \"kubernetes.io/projected/a033849b-0d61-42a5-9aa8-34f6b749b22f-kube-api-access-2mnhq\") pod \"monitoring-plugin-d5dcff7cf-wzr47\" (UID: \"a033849b-0d61-42a5-9aa8-34f6b749b22f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:47.728950 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.728925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:36:47.819814 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.819773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a033849b-0d61-42a5-9aa8-34f6b749b22f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wzr47\" (UID: \"a033849b-0d61-42a5-9aa8-34f6b749b22f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:47.820012 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.819847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnhq\" (UniqueName: \"kubernetes.io/projected/a033849b-0d61-42a5-9aa8-34f6b749b22f-kube-api-access-2mnhq\") pod \"monitoring-plugin-d5dcff7cf-wzr47\" (UID: \"a033849b-0d61-42a5-9aa8-34f6b749b22f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:47.820012 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:47.819925    2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 06 20:36:47.820112 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:36:47.820015    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a033849b-0d61-42a5-9aa8-34f6b749b22f-monitoring-plugin-cert podName:a033849b-0d61-42a5-9aa8-34f6b749b22f nodeName:}" failed. No retries permitted until 2026-07-06 20:36:48.319999833 +0000 UTC m=+105.654340414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a033849b-0d61-42a5-9aa8-34f6b749b22f-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-wzr47" (UID: "a033849b-0d61-42a5-9aa8-34f6b749b22f") : secret "monitoring-plugin-cert" not found
Jul 06 20:36:47.828982 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.828934    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnhq\" (UniqueName: \"kubernetes.io/projected/a033849b-0d61-42a5-9aa8-34f6b749b22f-kube-api-access-2mnhq\") pod \"monitoring-plugin-d5dcff7cf-wzr47\" (UID: \"a033849b-0d61-42a5-9aa8-34f6b749b22f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:47.992423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:47.992389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.030503 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.030477    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86bddcdcdf-lprv2"]
Jul 06 20:36:48.032281 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:48.032257    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4162691b_9b53_4f69_b26e_c10d6e157587.slice/crio-e94125b2806e6f9de80e8ea074ecee95b5feaa2360425143c20ec9bad501293b WatchSource:0}: Error finding container e94125b2806e6f9de80e8ea074ecee95b5feaa2360425143c20ec9bad501293b: Status 404 returned error can't find the container with id e94125b2806e6f9de80e8ea074ecee95b5feaa2360425143c20ec9bad501293b
Jul 06 20:36:48.098981 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.095647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.101433 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.101287    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-746978fcf7-tlq6x"]
Jul 06 20:36:48.122319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.122292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.124627 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.124603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-746978fcf7-tlq6x"]
Jul 06 20:36:48.126597 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.126577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 06 20:36:48.126678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.126602    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 06 20:36:48.126678 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.126644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 06 20:36:48.126894 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.126881    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 06 20:36:48.128039 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.128022    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 06 20:36:48.128145 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.128043    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-849tp\""
Jul 06 20:36:48.134660 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.134640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 06 20:36:48.192178 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.192154    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:48.223492 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.223587 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-metrics-client-ca\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.223645 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.223688 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-federate-client-tls\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.223723 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk95n\" (UniqueName: \"kubernetes.io/projected/1eb84500-930c-4c37-8942-6a27e39ff50c-kube-api-access-vk95n\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.223788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-secret-telemeter-client\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.223823 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-telemeter-client-tls\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.223823 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.223817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-serving-certs-ca-bundle\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.324997 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.324943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.324997 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.324998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-federate-client-tls\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.325171 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.325018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk95n\" (UniqueName: \"kubernetes.io/projected/1eb84500-930c-4c37-8942-6a27e39ff50c-kube-api-access-vk95n\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.325171 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.325079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-secret-telemeter-client\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.325171 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.325112    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-telemeter-client-tls\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.325171 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.325135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-serving-certs-ca-bundle\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.325171 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.325169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a033849b-0d61-42a5-9aa8-34f6b749b22f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wzr47\" (UID: \"a033849b-0d61-42a5-9aa8-34f6b749b22f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:48.325399 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.325213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.325399 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.325383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-metrics-client-ca\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.326077 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.326048    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-serving-certs-ca-bundle\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.326199 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.326059    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-metrics-client-ca\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.326199 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.326178    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb84500-930c-4c37-8942-6a27e39ff50c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.327783 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.327760    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-telemeter-client-tls\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.327910 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.327881    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.328079 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.328060    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-secret-telemeter-client\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.328214 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.328197    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a033849b-0d61-42a5-9aa8-34f6b749b22f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wzr47\" (UID: \"a033849b-0d61-42a5-9aa8-34f6b749b22f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:48.328348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.328333    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1eb84500-930c-4c37-8942-6a27e39ff50c-federate-client-tls\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.335046 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.335025    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk95n\" (UniqueName: \"kubernetes.io/projected/1eb84500-930c-4c37-8942-6a27e39ff50c-kube-api-access-vk95n\") pod \"telemeter-client-746978fcf7-tlq6x\" (UID: \"1eb84500-930c-4c37-8942-6a27e39ff50c\") " pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.438075 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.438043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x"
Jul 06 20:36:48.490802 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.490771    2582 patch_prober.go:28] interesting pod/image-registry-55cdb97ff5-g8zn2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 06 20:36:48.490939 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.490826    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2" podUID="73657f86-99d3-4f35-8bd4-a9fc7b56c977" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:36:48.506562 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.506504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:48.607694 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.607663    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-746978fcf7-tlq6x"]
Jul 06 20:36:48.610904 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:48.610875    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb84500_930c_4c37_8942_6a27e39ff50c.slice/crio-f4b4588bee615f84aaab24f3096d5f7fa9c14d1ad29d4c7b6e314b51752c0011 WatchSource:0}: Error finding container f4b4588bee615f84aaab24f3096d5f7fa9c14d1ad29d4c7b6e314b51752c0011: Status 404 returned error can't find the container with id f4b4588bee615f84aaab24f3096d5f7fa9c14d1ad29d4c7b6e314b51752c0011
Jul 06 20:36:48.665844 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.665671    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"]
Jul 06 20:36:48.668398 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:36:48.668375    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda033849b_0d61_42a5_9aa8_34f6b749b22f.slice/crio-37826f0f59dd6446af3b46465b330f9ee456d0675abc7681e2bb1b6da3722ee6 WatchSource:0}: Error finding container 37826f0f59dd6446af3b46465b330f9ee456d0675abc7681e2bb1b6da3722ee6: Status 404 returned error can't find the container with id 37826f0f59dd6446af3b46465b330f9ee456d0675abc7681e2bb1b6da3722ee6
Jul 06 20:36:48.843919 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.843833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" event={"ID":"fa0ab60a-9947-406c-9495-7f4cf0512c8b","Type":"ContainerStarted","Data":"1e3064d11b13990e670cd83e1c2bc29e529b40f8012634cd8317cbb483774de5"}
Jul 06 20:36:48.844108 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.843946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" event={"ID":"fa0ab60a-9947-406c-9495-7f4cf0512c8b","Type":"ContainerStarted","Data":"9a61c61a9bfb31b59dbb45f1ff4d9784498b36d816ea00b8ef7887e293c7178c"}
Jul 06 20:36:48.844108 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.844009    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" event={"ID":"fa0ab60a-9947-406c-9495-7f4cf0512c8b","Type":"ContainerStarted","Data":"dca76141728d378dae6f75de27d6b6988277b7ad5bc6feeda4eeb422713d4a99"}
Jul 06 20:36:48.845151 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.845124    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47" event={"ID":"a033849b-0d61-42a5-9aa8-34f6b749b22f","Type":"ContainerStarted","Data":"37826f0f59dd6446af3b46465b330f9ee456d0675abc7681e2bb1b6da3722ee6"}
Jul 06 20:36:48.846392 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.846365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x" event={"ID":"1eb84500-930c-4c37-8942-6a27e39ff50c","Type":"ContainerStarted","Data":"f4b4588bee615f84aaab24f3096d5f7fa9c14d1ad29d4c7b6e314b51752c0011"}
Jul 06 20:36:48.847600 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:48.847580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2" event={"ID":"4162691b-9b53-4f69-b26e-c10d6e157587","Type":"ContainerStarted","Data":"e94125b2806e6f9de80e8ea074ecee95b5feaa2360425143c20ec9bad501293b"}
Jul 06 20:36:49.051271 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.051236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:49.341386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.341353    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:49.443865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.443788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:49.863830 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.863790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2" event={"ID":"4162691b-9b53-4f69-b26e-c10d6e157587","Type":"ContainerStarted","Data":"fb030dba77cdb5de0eb071ebab9a00e23ba0907248612775cb7fc93d9eb6c380"}
Jul 06 20:36:49.870131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.870094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" event={"ID":"fa0ab60a-9947-406c-9495-7f4cf0512c8b","Type":"ContainerStarted","Data":"0d79af8e908ee077d9f28d0ffadcd29e1e513f249c63729632da174c176a379f"}
Jul 06 20:36:49.870279 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.870164    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" event={"ID":"fa0ab60a-9947-406c-9495-7f4cf0512c8b","Type":"ContainerStarted","Data":"9dc6f81dbcb7bda3107ab6f0459c97f106ce76afc718be32ace28df5870bbda6"}
Jul 06 20:36:49.870279 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.870211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" event={"ID":"fa0ab60a-9947-406c-9495-7f4cf0512c8b","Type":"ContainerStarted","Data":"bef330f0995e98a32c04713316e82d1c451411894a52eba2edcf658388b70428"}
Jul 06 20:36:49.870279 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.870258    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:49.885634 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.885583    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2" podStartSLOduration=1.198713814 podStartE2EDuration="2.885567932s" totalImagesPullingTime="1.686854118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:49.883436113 +0000 UTC m=+107.217776705" watchObservedRunningTime="2026-07-06 20:36:49.885567932 +0000 UTC m=+107.219908527"
Jul 06 20:36:49.911567 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.911510    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss" podStartSLOduration=2.518627738 podStartE2EDuration="5.911495886s" totalImagesPullingTime="3.392868148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:49.909443756 +0000 UTC m=+107.243784391" watchObservedRunningTime="2026-07-06 20:36:49.911495886 +0000 UTC m=+107.245836481"
Jul 06 20:36:49.917096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:49.917070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:50.015178 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.015094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:50.749310 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.749286    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55cdb97ff5-g8zn2"
Jul 06 20:36:50.824502 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.824402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:50.875710 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.875680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47" event={"ID":"a033849b-0d61-42a5-9aa8-34f6b749b22f","Type":"ContainerStarted","Data":"23290a2d578b9cfa5c4a0254c606cfee39c310b881e46cabbc9aca99bdef5e46"}
Jul 06 20:36:50.875918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.875879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:50.877535 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.877504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x" event={"ID":"1eb84500-930c-4c37-8942-6a27e39ff50c","Type":"ContainerStarted","Data":"8eeeaa78f8ce98a37c283005553cd3ad348c09b2eb13642ba3eebe4c5ea8c8df"}
Jul 06 20:36:50.882013 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.881993    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47"
Jul 06 20:36:50.949847 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.949794    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wzr47" podStartSLOduration=1.891862765 podStartE2EDuration="3.949778627s" totalImagesPullingTime="2.057915862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:50.947265597 +0000 UTC m=+108.281606192" watchObservedRunningTime="2026-07-06 20:36:50.949778627 +0000 UTC m=+108.284119221"
Jul 06 20:36:50.996310 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:50.996239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:51.095153 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:51.095122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:51.883907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:51.883804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x" event={"ID":"1eb84500-930c-4c37-8942-6a27e39ff50c","Type":"ContainerStarted","Data":"c22f347531c1652caee15afdc9bbde8f4b24c614403c1978d02d9040bd11d4af"}
Jul 06 20:36:51.883907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:51.883863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x" event={"ID":"1eb84500-930c-4c37-8942-6a27e39ff50c","Type":"ContainerStarted","Data":"50222819a5ce7aca48cbe98ee4de4592415f28ac46e414b89201fc52ca25ae3b"}
Jul 06 20:36:51.913020 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:51.912956    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-746978fcf7-tlq6x" podStartSLOduration=0.990901632 podStartE2EDuration="3.912938396s" totalImagesPullingTime="2.922036764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:51.912125441 +0000 UTC m=+109.246466036" watchObservedRunningTime="2026-07-06 20:36:51.912938396 +0000 UTC m=+109.247278991"
Jul 06 20:36:53.293892 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:53.293856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:53.392789 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:53.392755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:54.821359 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:54.821332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rps8j"
Jul 06 20:36:54.995483 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:54.995450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:55.097166 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:55.097081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:55.883379 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:55.883352    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-687687fc7d-drjss"
Jul 06 20:36:58.145577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:58.145539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:58.280399 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:58.280368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:36:58.407742 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:36:58.407660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:00.154334 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:00.154267    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:00.259223 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:00.259190    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:01.317555 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:01.317522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:01.419534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:01.419496    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:01.518638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:01.518613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:02.392309 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:02.392276    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:02.454413 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:02.454382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:03.749817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:03.749785    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff9c8fd7b-kkh6q"]
Jul 06 20:37:03.883844 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:03.883806    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:03.982032 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:03.981992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:04.174055 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:04.173953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:04.273820 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:04.273790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:06.423485 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:06.423455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:06.523718 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:06.523681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:06.630619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:06.630580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:07.729768 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:07.729733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:37:07.730179 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:07.729792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:37:08.847487 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:08.847449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:08.911098 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:08.911048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:09.134545 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:09.134473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:09.203020 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:09.202991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:11.954931 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:11.954884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:12.022610 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:12.022576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:12.935114 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:12.935083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:37:12.937309 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:12.937280    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ec3ed5-d7c9-4842-a336-9579edc57bf4-metrics-certs\") pod \"network-metrics-daemon-j8bqp\" (UID: \"c0ec3ed5-d7c9-4842-a336-9579edc57bf4\") " pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:37:12.946445 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:12.946415    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl572\""
Jul 06 20:37:12.954019 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:12.953999    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j8bqp"
Jul 06 20:37:13.101809 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:13.101782    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j8bqp"]
Jul 06 20:37:13.104318 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:37:13.104291    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ec3ed5_d7c9_4842_a336_9579edc57bf4.slice/crio-3c415915ee397e8ebb50ed5d96895728ebb0b4af3266a25b8c4a056dd7164988 WatchSource:0}: Error finding container 3c415915ee397e8ebb50ed5d96895728ebb0b4af3266a25b8c4a056dd7164988: Status 404 returned error can't find the container with id 3c415915ee397e8ebb50ed5d96895728ebb0b4af3266a25b8c4a056dd7164988
Jul 06 20:37:13.964577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:13.964537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j8bqp" event={"ID":"c0ec3ed5-d7c9-4842-a336-9579edc57bf4","Type":"ContainerStarted","Data":"3c415915ee397e8ebb50ed5d96895728ebb0b4af3266a25b8c4a056dd7164988"}
Jul 06 20:37:14.970207 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:14.970172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j8bqp" event={"ID":"c0ec3ed5-d7c9-4842-a336-9579edc57bf4","Type":"ContainerStarted","Data":"bddbf52e582f7139699a13d409dc476ef7dc495f38b8dab5df7286fa299deae5"}
Jul 06 20:37:14.970599 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:14.970224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j8bqp" event={"ID":"c0ec3ed5-d7c9-4842-a336-9579edc57bf4","Type":"ContainerStarted","Data":"eab783ffb0676a69789c7f91847110f27b0ca329fd75a4b64df081451cef927b"}
Jul 06 20:37:14.987769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:14.987715    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j8bqp" podStartSLOduration=131.011165762 podStartE2EDuration="2m11.987700385s" totalImagesPullingTime="976.534623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:14.986905295 +0000 UTC m=+132.321245905" watchObservedRunningTime="2026-07-06 20:37:14.987700385 +0000 UTC m=+132.322040980"
Jul 06 20:37:15.091317 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:15.091286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:15.151242 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:15.151209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:26.950212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:26.950179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:27.053259 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:27.053225    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:27.734794 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:27.734755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:37:27.738478 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:27.738460    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86bddcdcdf-lprv2"
Jul 06 20:37:28.771303 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:28.771247    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5ff9c8fd7b-kkh6q" podUID="16b1624a-3cac-49a2-b63f-acbe45d9071c" containerName="console" containerID="cri-o://344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307" gracePeriod=15
Jul 06 20:37:29.017660 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.017635    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:37:29.021542 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.021478    2582 generic.go:356] "Generic (PLEG): container finished" podID="16b1624a-3cac-49a2-b63f-acbe45d9071c" containerID="344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307" exitCode=0
Jul 06 20:37:29.021542 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.021518    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff9c8fd7b-kkh6q"
Jul 06 20:37:29.021720 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.021549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9c8fd7b-kkh6q" event={"ID":"16b1624a-3cac-49a2-b63f-acbe45d9071c","Type":"ContainerDied","Data":"344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307"}
Jul 06 20:37:29.021720 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.021593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff9c8fd7b-kkh6q" event={"ID":"16b1624a-3cac-49a2-b63f-acbe45d9071c","Type":"ContainerDied","Data":"0cfa6262f4352ccaed76f9f50fac55112fca03cf59a8cdc1b7e0d9bc33abe956"}
Jul 06 20:37:29.021720 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.021617    2582 scope.go:117] "RemoveContainer" containerID="344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307"
Jul 06 20:37:29.028947 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.028930    2582 scope.go:117] "RemoveContainer" containerID="344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307"
Jul 06 20:37:29.029243 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:37:29.029223    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307\": container with ID starting with 344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307 not found: ID does not exist" containerID="344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307"
Jul 06 20:37:29.029314 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.029255    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307"} err="failed to get container status \"344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307\": rpc error: code = NotFound desc = could not find container \"344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307\": container with ID starting with 344c616e4a651baa7d94adf83f85a7af42b74c57061118ede14b0b3066eb9307 not found: ID does not exist"
Jul 06 20:37:29.168074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168043    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-config\") pod \"16b1624a-3cac-49a2-b63f-acbe45d9071c\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") "
Jul 06 20:37:29.168258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168091    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-serving-cert\") pod \"16b1624a-3cac-49a2-b63f-acbe45d9071c\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") "
Jul 06 20:37:29.168258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-oauth-config\") pod \"16b1624a-3cac-49a2-b63f-acbe45d9071c\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") "
Jul 06 20:37:29.168258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-service-ca\") pod \"16b1624a-3cac-49a2-b63f-acbe45d9071c\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") "
Jul 06 20:37:29.168258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168187    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-oauth-serving-cert\") pod \"16b1624a-3cac-49a2-b63f-acbe45d9071c\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") "
Jul 06 20:37:29.168258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168204    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-trusted-ca-bundle\") pod \"16b1624a-3cac-49a2-b63f-acbe45d9071c\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") "
Jul 06 20:37:29.168258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs87d\" (UniqueName: \"kubernetes.io/projected/16b1624a-3cac-49a2-b63f-acbe45d9071c-kube-api-access-xs87d\") pod \"16b1624a-3cac-49a2-b63f-acbe45d9071c\" (UID: \"16b1624a-3cac-49a2-b63f-acbe45d9071c\") "
Jul 06 20:37:29.168579 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168544    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-config" (OuterVolumeSpecName: "console-config") pod "16b1624a-3cac-49a2-b63f-acbe45d9071c" (UID: "16b1624a-3cac-49a2-b63f-acbe45d9071c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:29.168718 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168693    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-service-ca" (OuterVolumeSpecName: "service-ca") pod "16b1624a-3cac-49a2-b63f-acbe45d9071c" (UID: "16b1624a-3cac-49a2-b63f-acbe45d9071c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:29.168804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168723    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "16b1624a-3cac-49a2-b63f-acbe45d9071c" (UID: "16b1624a-3cac-49a2-b63f-acbe45d9071c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:29.168859 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.168804    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "16b1624a-3cac-49a2-b63f-acbe45d9071c" (UID: "16b1624a-3cac-49a2-b63f-acbe45d9071c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:37:29.170416 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.170392    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "16b1624a-3cac-49a2-b63f-acbe45d9071c" (UID: "16b1624a-3cac-49a2-b63f-acbe45d9071c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:37:29.170506 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.170415    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "16b1624a-3cac-49a2-b63f-acbe45d9071c" (UID: "16b1624a-3cac-49a2-b63f-acbe45d9071c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:37:29.170563 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.170537    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b1624a-3cac-49a2-b63f-acbe45d9071c-kube-api-access-xs87d" (OuterVolumeSpecName: "kube-api-access-xs87d") pod "16b1624a-3cac-49a2-b63f-acbe45d9071c" (UID: "16b1624a-3cac-49a2-b63f-acbe45d9071c"). InnerVolumeSpecName "kube-api-access-xs87d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:37:29.269170 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.269133    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-oauth-config\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:37:29.269170 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.269159    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:37:29.269170 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.269170    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-oauth-serving-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:37:29.269170 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.269180    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-trusted-ca-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:37:29.269426 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.269189    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs87d\" (UniqueName: \"kubernetes.io/projected/16b1624a-3cac-49a2-b63f-acbe45d9071c-kube-api-access-xs87d\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:37:29.269426 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.269199    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-config\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:37:29.269426 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.269207    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16b1624a-3cac-49a2-b63f-acbe45d9071c-console-serving-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:37:29.344424 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.344386    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff9c8fd7b-kkh6q"]
Jul 06 20:37:29.348281 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:29.348255    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff9c8fd7b-kkh6q"]
Jul 06 20:37:31.338944 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:31.338902    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b1624a-3cac-49a2-b63f-acbe45d9071c" path="/var/lib/kubelet/pods/16b1624a-3cac-49a2-b63f-acbe45d9071c/volumes"
Jul 06 20:37:58.936191 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:58.936108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:37:59.056604 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:37:59.056570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:03.002429 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:03.002397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:05.498154 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.498119    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:38:05.498523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.498426    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b1624a-3cac-49a2-b63f-acbe45d9071c" containerName="console"
Jul 06 20:38:05.498523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.498439    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b1624a-3cac-49a2-b63f-acbe45d9071c" containerName="console"
Jul 06 20:38:05.498523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.498505    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="16b1624a-3cac-49a2-b63f-acbe45d9071c" containerName="console"
Jul 06 20:38:05.503258 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.503242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.506665 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.506641    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 06 20:38:05.506783 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.506644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 06 20:38:05.506783 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.506688    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 06 20:38:05.506941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.506928    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 06 20:38:05.507027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.507009    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 06 20:38:05.507078 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.507031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6r66r\""
Jul 06 20:38:05.507114 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.507096    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 06 20:38:05.507364 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.507348    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 06 20:38:05.507453 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.507439    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 06 20:38:05.514193 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.514174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 06 20:38:05.518274 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.518255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:38:05.547896 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.547867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de04391c-d7e4-4837-b02d-e228888e79e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548037 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.547909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548037 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.547939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548037 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.547994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs97g\" (UniqueName: \"kubernetes.io/projected/de04391c-d7e4-4837-b02d-e228888e79e5-kube-api-access-xs97g\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548037 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de04391c-d7e4-4837-b02d-e228888e79e5-config-out\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548176 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548176 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de04391c-d7e4-4837-b02d-e228888e79e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548176 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de04391c-d7e4-4837-b02d-e228888e79e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548275 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548275 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-web-config\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548275 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548275 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.548444 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.548345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de04391c-d7e4-4837-b02d-e228888e79e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649160 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649160 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649408 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs97g\" (UniqueName: \"kubernetes.io/projected/de04391c-d7e4-4837-b02d-e228888e79e5-kube-api-access-xs97g\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649408 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de04391c-d7e4-4837-b02d-e228888e79e5-config-out\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649408 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649408 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de04391c-d7e4-4837-b02d-e228888e79e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de04391c-d7e4-4837-b02d-e228888e79e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.649865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649842    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de04391c-d7e4-4837-b02d-e228888e79e5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.650006 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.649944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.650226 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.650208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-web-config\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.650418 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.650302    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de04391c-d7e4-4837-b02d-e228888e79e5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.650506 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.650401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.650506 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.650481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.650618 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.650549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de04391c-d7e4-4837-b02d-e228888e79e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.650618 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.650582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de04391c-d7e4-4837-b02d-e228888e79e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.652074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.652049    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de04391c-d7e4-4837-b02d-e228888e79e5-config-out\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.652348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.652320    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de04391c-d7e4-4837-b02d-e228888e79e5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.652439 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.652353    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.652845 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.652824    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.653006 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.652848    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.653092 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.653039    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-web-config\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.653272 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.653255    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.653381 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.653357    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.653902 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.653881    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de04391c-d7e4-4837-b02d-e228888e79e5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.654523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.654506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de04391c-d7e4-4837-b02d-e228888e79e5-config-volume\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.658034 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.658014    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs97g\" (UniqueName: \"kubernetes.io/projected/de04391c-d7e4-4837-b02d-e228888e79e5-kube-api-access-xs97g\") pod \"alertmanager-main-0\" (UID: \"de04391c-d7e4-4837-b02d-e228888e79e5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.813261 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.813174    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 06 20:38:05.959287 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:05.959263    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 06 20:38:05.961179 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:38:05.961145    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde04391c_d7e4_4837_b02d_e228888e79e5.slice/crio-463c9651f529c0027b24a7173c5c282fce294bf29cdce1fb9caa01ffe1edb270 WatchSource:0}: Error finding container 463c9651f529c0027b24a7173c5c282fce294bf29cdce1fb9caa01ffe1edb270: Status 404 returned error can't find the container with id 463c9651f529c0027b24a7173c5c282fce294bf29cdce1fb9caa01ffe1edb270
Jul 06 20:38:06.170304 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:06.170206    2582 generic.go:356] "Generic (PLEG): container finished" podID="de04391c-d7e4-4837-b02d-e228888e79e5" containerID="cbe03b2a4499d26ecef87fd66c94ad46c020bd6a89df5d46481d5699d76f39cd" exitCode=0
Jul 06 20:38:06.170474 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:06.170297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerDied","Data":"cbe03b2a4499d26ecef87fd66c94ad46c020bd6a89df5d46481d5699d76f39cd"}
Jul 06 20:38:06.170474 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:06.170388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerStarted","Data":"463c9651f529c0027b24a7173c5c282fce294bf29cdce1fb9caa01ffe1edb270"}
Jul 06 20:38:08.180886 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:08.180851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerStarted","Data":"f251017896dce939b1cbef5f6f8e1999bd69cfb45c87c6166f802efde6ca3b57"}
Jul 06 20:38:08.181388 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:08.180925    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerStarted","Data":"8ba7b820304d89c812a15fb37043b664a69e60f740b502f75e1dba7688648d2b"}
Jul 06 20:38:08.181388 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:08.180993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerStarted","Data":"aeb430e6d2a8ab544327fb99b5d49271e3ed0b5fb4f2a22b6d44490665347e5e"}
Jul 06 20:38:08.181388 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:08.181039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerStarted","Data":"71d944ef72d81f0290f334bc81ce5aa144734ec282f3b80ae8d0f21987b92889"}
Jul 06 20:38:08.181388 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:08.181085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerStarted","Data":"7dd929f30d411ebf6e756ff25be480ced409b993c1e7f6d073f0e26c306fa21d"}
Jul 06 20:38:08.181388 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:08.181123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de04391c-d7e4-4837-b02d-e228888e79e5","Type":"ContainerStarted","Data":"55c09e994ca694a581296162b597ea668e54297a441d3c74c4e3db304d547a95"}
Jul 06 20:38:08.217181 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:08.217120    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.024478339 podStartE2EDuration="3.217103281s" totalImagesPullingTime="1.176796586s" totalInitContainerRuntime="15.828356ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:08.215836515 +0000 UTC m=+185.550177111" watchObservedRunningTime="2026-07-06 20:38:08.217103281 +0000 UTC m=+185.551443876"
Jul 06 20:38:10.821701 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:10.821665    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:10.883619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:10.883591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:18.268483 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:18.268450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:18.367129 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:18.367098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:18.450152 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:18.450118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:18.977687 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:18.977649    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:19.078276 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:19.078243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:26.968184 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:26.968147    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:27.068656 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:27.068625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:29.182296 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:29.182264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:29.243768 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:29.243724    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:29.299162 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:29.299130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:41.404126 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:41.404082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:49.868740 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:49.868709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:49.946949 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:49.946919    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:53.911527 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:53.911493    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:38:53.979210 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:38:53.979177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:04.990872 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:04.990830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:05.049372 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:05.049344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:05.356627 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:05.356542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:05.457238 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:05.457197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:05.527226 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:05.527193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:06.482573 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.482543    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"]
Jul 06 20:39:06.485830 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.485815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.489093 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.489070    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 06 20:39:06.489199 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.489136    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 06 20:39:06.490035 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.490021    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r744q\""
Jul 06 20:39:06.496557 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.496539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"]
Jul 06 20:39:06.543883 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.543854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjptq\" (UniqueName: \"kubernetes.io/projected/3def2979-47d5-4b24-9472-96bf8bc9a46e-kube-api-access-fjptq\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.543883 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.543884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.544112 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.544034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.645337 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.645297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.645496 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.645360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjptq\" (UniqueName: \"kubernetes.io/projected/3def2979-47d5-4b24-9472-96bf8bc9a46e-kube-api-access-fjptq\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.645496 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.645389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.645779 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.645759    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.645779 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.645773    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.654433 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.654403    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjptq\" (UniqueName: \"kubernetes.io/projected/3def2979-47d5-4b24-9472-96bf8bc9a46e-kube-api-access-fjptq\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.795662 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.795576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:06.929222 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:06.929198    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"]
Jul 06 20:39:06.931569 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:39:06.931537    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3def2979_47d5_4b24_9472_96bf8bc9a46e.slice/crio-8b57d28fd5239dab083a1fab5f515b8d52f01a1bd5f3b7150598e8451883b754 WatchSource:0}: Error finding container 8b57d28fd5239dab083a1fab5f515b8d52f01a1bd5f3b7150598e8451883b754: Status 404 returned error can't find the container with id 8b57d28fd5239dab083a1fab5f515b8d52f01a1bd5f3b7150598e8451883b754
Jul 06 20:39:07.407861 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:07.407825    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr" event={"ID":"3def2979-47d5-4b24-9472-96bf8bc9a46e","Type":"ContainerStarted","Data":"8b57d28fd5239dab083a1fab5f515b8d52f01a1bd5f3b7150598e8451883b754"}
Jul 06 20:39:15.443410 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:15.443371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr" event={"ID":"3def2979-47d5-4b24-9472-96bf8bc9a46e","Type":"ContainerStarted","Data":"4ef7fea716dc78f0ad2b3847f5cb54d006866675dcd68a45ad4b6bd352bfd944"}
Jul 06 20:39:16.449630 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:16.449594    2582 generic.go:356] "Generic (PLEG): container finished" podID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerID="4ef7fea716dc78f0ad2b3847f5cb54d006866675dcd68a45ad4b6bd352bfd944" exitCode=0
Jul 06 20:39:16.450018 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:16.449669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr" event={"ID":"3def2979-47d5-4b24-9472-96bf8bc9a46e","Type":"ContainerDied","Data":"4ef7fea716dc78f0ad2b3847f5cb54d006866675dcd68a45ad4b6bd352bfd944"}
Jul 06 20:39:19.464041 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:19.464008    2582 generic.go:356] "Generic (PLEG): container finished" podID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerID="dc7676c7dc63c2c323e08f8a41c496885ee4aad5231a5494769c062563191c1b" exitCode=0
Jul 06 20:39:19.464409 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:19.464073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr" event={"ID":"3def2979-47d5-4b24-9472-96bf8bc9a46e","Type":"ContainerDied","Data":"dc7676c7dc63c2c323e08f8a41c496885ee4aad5231a5494769c062563191c1b"}
Jul 06 20:39:27.532674 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:27.532574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:27.601886 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:27.601851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:32.521471 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:32.521435    2582 generic.go:356] "Generic (PLEG): container finished" podID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerID="46cdfea486e24a5cca89cc0cd214cf4ba8abc83ccdee570d414f3ba6d351c295" exitCode=0
Jul 06 20:39:32.521862 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:32.521527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr" event={"ID":"3def2979-47d5-4b24-9472-96bf8bc9a46e","Type":"ContainerDied","Data":"46cdfea486e24a5cca89cc0cd214cf4ba8abc83ccdee570d414f3ba6d351c295"}
Jul 06 20:39:33.647415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.647393    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:33.797683 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.797608    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-bundle\") pod \"3def2979-47d5-4b24-9472-96bf8bc9a46e\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") "
Jul 06 20:39:33.797683 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.797648    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjptq\" (UniqueName: \"kubernetes.io/projected/3def2979-47d5-4b24-9472-96bf8bc9a46e-kube-api-access-fjptq\") pod \"3def2979-47d5-4b24-9472-96bf8bc9a46e\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") "
Jul 06 20:39:33.797862 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.797693    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-util\") pod \"3def2979-47d5-4b24-9472-96bf8bc9a46e\" (UID: \"3def2979-47d5-4b24-9472-96bf8bc9a46e\") "
Jul 06 20:39:33.798255 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.798217    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-bundle" (OuterVolumeSpecName: "bundle") pod "3def2979-47d5-4b24-9472-96bf8bc9a46e" (UID: "3def2979-47d5-4b24-9472-96bf8bc9a46e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:39:33.799825 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.799803    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3def2979-47d5-4b24-9472-96bf8bc9a46e-kube-api-access-fjptq" (OuterVolumeSpecName: "kube-api-access-fjptq") pod "3def2979-47d5-4b24-9472-96bf8bc9a46e" (UID: "3def2979-47d5-4b24-9472-96bf8bc9a46e"). InnerVolumeSpecName "kube-api-access-fjptq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:39:33.802424 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.802405    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-util" (OuterVolumeSpecName: "util") pod "3def2979-47d5-4b24-9472-96bf8bc9a46e" (UID: "3def2979-47d5-4b24-9472-96bf8bc9a46e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:39:33.898651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.898617    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:39:33.898651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.898644    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3def2979-47d5-4b24-9472-96bf8bc9a46e-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:39:33.898651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:33.898653    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjptq\" (UniqueName: \"kubernetes.io/projected/3def2979-47d5-4b24-9472-96bf8bc9a46e-kube-api-access-fjptq\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:39:34.531223 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:34.531193    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr"
Jul 06 20:39:34.531223 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:34.531203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589g5rsr" event={"ID":"3def2979-47d5-4b24-9472-96bf8bc9a46e","Type":"ContainerDied","Data":"8b57d28fd5239dab083a1fab5f515b8d52f01a1bd5f3b7150598e8451883b754"}
Jul 06 20:39:34.531432 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:34.531259    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b57d28fd5239dab083a1fab5f515b8d52f01a1bd5f3b7150598e8451883b754"
Jul 06 20:39:38.437390 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:38.437359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:38.533610 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:38.533578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:44.515365 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.515327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:44.818775 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.818682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:44.830357 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830328    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-jl7h5"]
Jul 06 20:39:44.830653 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830640    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerName="pull"
Jul 06 20:39:44.830699 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830654    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerName="pull"
Jul 06 20:39:44.830699 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830670    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerName="util"
Jul 06 20:39:44.830699 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830675    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerName="util"
Jul 06 20:39:44.830699 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerName="extract"
Jul 06 20:39:44.830699 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830697    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerName="extract"
Jul 06 20:39:44.830850 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.830749    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3def2979-47d5-4b24-9472-96bf8bc9a46e" containerName="extract"
Jul 06 20:39:44.832680 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.832665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:44.836339 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.836319    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 06 20:39:44.836339 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.836336    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 06 20:39:44.836339 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.836338    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 06 20:39:44.836566 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.836324    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 06 20:39:44.837672 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.837655    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tvqxx\""
Jul 06 20:39:44.837731 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.837657    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 06 20:39:44.846541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.846522    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-jl7h5"]
Jul 06 20:39:44.891578 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.891544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-certificates\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:44.891578 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.891582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6mf\" (UniqueName: \"kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-kube-api-access-sx6mf\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:44.891771 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.891700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3dcbb06f-2f9d-4026-903a-da6a0b2905af-cabundle0\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:44.992283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.992233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3dcbb06f-2f9d-4026-903a-da6a0b2905af-cabundle0\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:44.992283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.992299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-certificates\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:44.992534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:44.992319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6mf\" (UniqueName: \"kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-kube-api-access-sx6mf\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:44.992534 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:39:44.992419    2582 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 06 20:39:44.992534 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:39:44.992442    2582 secret.go:281] references non-existent secret key: ca.crt
Jul 06 20:39:44.992534 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:39:44.992443    2582 configmap.go:193] Couldn't get configMap openshift-keda/keda-ocp-cabundle: configmap "keda-ocp-cabundle" not found
Jul 06 20:39:44.992534 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:39:44.992449    2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 06 20:39:44.992764 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:39:44.992546    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3dcbb06f-2f9d-4026-903a-da6a0b2905af-cabundle0 podName:3dcbb06f-2f9d-4026-903a-da6a0b2905af nodeName:}" failed. No retries permitted until 2026-07-06 20:39:45.492524684 +0000 UTC m=+282.826865259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cabundle0" (UniqueName: "kubernetes.io/configmap/3dcbb06f-2f9d-4026-903a-da6a0b2905af-cabundle0") pod "keda-operator-68bff59c-jl7h5" (UID: "3dcbb06f-2f9d-4026-903a-da6a0b2905af") : configmap "keda-ocp-cabundle" not found
Jul 06 20:39:44.992764 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:39:44.992553    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-jl7h5: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 06 20:39:44.992764 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:39:44.992607    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-certificates podName:3dcbb06f-2f9d-4026-903a-da6a0b2905af nodeName:}" failed. No retries permitted until 2026-07-06 20:39:45.492593337 +0000 UTC m=+282.826933913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-certificates") pod "keda-operator-68bff59c-jl7h5" (UID: "3dcbb06f-2f9d-4026-903a-da6a0b2905af") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 06 20:39:45.001363 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.001333    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6mf\" (UniqueName: \"kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-kube-api-access-sx6mf\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:45.401057 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.401020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:45.496091 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.496049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3dcbb06f-2f9d-4026-903a-da6a0b2905af-cabundle0\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:45.496259 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.496106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-certificates\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:45.496668 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.496647    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3dcbb06f-2f9d-4026-903a-da6a0b2905af-cabundle0\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:45.498604 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.498583    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcbb06f-2f9d-4026-903a-da6a0b2905af-certificates\") pod \"keda-operator-68bff59c-jl7h5\" (UID: \"3dcbb06f-2f9d-4026-903a-da6a0b2905af\") " pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:45.602251 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.602214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:45.742974 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.742936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:45.760194 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.760168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:45.894439 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.894414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-jl7h5"]
Jul 06 20:39:45.896578 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:39:45.896553    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcbb06f_2f9d_4026_903a_da6a0b2905af.slice/crio-2ac9e3e281fdf4f94c7a82f57c379140c26d9438ae3daafaa40543f9f5db89c8 WatchSource:0}: Error finding container 2ac9e3e281fdf4f94c7a82f57c379140c26d9438ae3daafaa40543f9f5db89c8: Status 404 returned error can't find the container with id 2ac9e3e281fdf4f94c7a82f57c379140c26d9438ae3daafaa40543f9f5db89c8
Jul 06 20:39:45.975423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:45.975379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:46.182824 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:46.182792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:46.281571 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:46.281543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:46.563301 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:46.563219    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:46.583818 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:46.583785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-jl7h5" event={"ID":"3dcbb06f-2f9d-4026-903a-da6a0b2905af","Type":"ContainerStarted","Data":"2ac9e3e281fdf4f94c7a82f57c379140c26d9438ae3daafaa40543f9f5db89c8"}
Jul 06 20:39:46.713883 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:46.713834    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:46.816428 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:46.816341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:39:48.592698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:48.592666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-jl7h5" event={"ID":"3dcbb06f-2f9d-4026-903a-da6a0b2905af","Type":"ContainerStarted","Data":"9b7710a681021e470f2996066a670395f5050150f37cf70fb800c0b997d4a16b"}
Jul 06 20:39:48.593044 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:48.592806    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:39:48.614693 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:39:48.614639    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-jl7h5" podStartSLOduration=1.9847570989999999 podStartE2EDuration="4.614624678s" totalImagesPullingTime="2.629867579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:48.612122315 +0000 UTC m=+285.946462911" watchObservedRunningTime="2026-07-06 20:39:48.614624678 +0000 UTC m=+285.948965273"
Jul 06 20:40:03.078008 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:03.077972    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:40:03.088568 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:03.088534    2582 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 20:40:03.089668 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:03.089642    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:40:09.599247 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:09.599208    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-jl7h5"
Jul 06 20:40:12.447217 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:12.447183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:12.548260 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:12.548228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:52.176354 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.176321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:52.234996 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.234945    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-2k89z"]
Jul 06 20:40:52.237173 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.237156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:52.240108 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.240086    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 06 20:40:52.240238 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.240133    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 06 20:40:52.241499 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.241474    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 06 20:40:52.241619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.241598    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-6j72w\""
Jul 06 20:40:52.251336 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.251313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-2k89z"]
Jul 06 20:40:52.260879 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.260854    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7f65c45d5c-7t766"]
Jul 06 20:40:52.263526 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.263507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.266366 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.266337    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 06 20:40:52.266467 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.266416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xnw54\""
Jul 06 20:40:52.276322 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.276295    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7f65c45d5c-7t766"]
Jul 06 20:40:52.348438 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.348404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m5bl\" (UniqueName: \"kubernetes.io/projected/2040c113-44c2-40c4-acf0-0cc9749c489a-kube-api-access-7m5bl\") pod \"llmisvc-controller-manager-7f65c45d5c-7t766\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") " pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.348438 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.348443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert\") pod \"kserve-controller-manager-68bc99fcb8-2k89z\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") " pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:52.348642 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.348506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzltf\" (UniqueName: \"kubernetes.io/projected/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-kube-api-access-fzltf\") pod \"kserve-controller-manager-68bc99fcb8-2k89z\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") " pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:52.348642 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.348542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2040c113-44c2-40c4-acf0-0cc9749c489a-cert\") pod \"llmisvc-controller-manager-7f65c45d5c-7t766\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") " pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.449879 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.449800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzltf\" (UniqueName: \"kubernetes.io/projected/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-kube-api-access-fzltf\") pod \"kserve-controller-manager-68bc99fcb8-2k89z\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") " pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:52.449879 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.449840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2040c113-44c2-40c4-acf0-0cc9749c489a-cert\") pod \"llmisvc-controller-manager-7f65c45d5c-7t766\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") " pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.450089 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.449895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m5bl\" (UniqueName: \"kubernetes.io/projected/2040c113-44c2-40c4-acf0-0cc9749c489a-kube-api-access-7m5bl\") pod \"llmisvc-controller-manager-7f65c45d5c-7t766\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") " pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.450089 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.449923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert\") pod \"kserve-controller-manager-68bc99fcb8-2k89z\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") " pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:52.450089 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:40:52.450047    2582 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 06 20:40:52.450196 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:40:52.450108    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert podName:712eeea6-45b6-4fd4-bdcd-9a12b87d01fe nodeName:}" failed. No retries permitted until 2026-07-06 20:40:52.950091423 +0000 UTC m=+350.284431997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert") pod "kserve-controller-manager-68bc99fcb8-2k89z" (UID: "712eeea6-45b6-4fd4-bdcd-9a12b87d01fe") : secret "kserve-webhook-server-cert" not found
Jul 06 20:40:52.452336 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.452319    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2040c113-44c2-40c4-acf0-0cc9749c489a-cert\") pod \"llmisvc-controller-manager-7f65c45d5c-7t766\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") " pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.467554 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.467524    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m5bl\" (UniqueName: \"kubernetes.io/projected/2040c113-44c2-40c4-acf0-0cc9749c489a-kube-api-access-7m5bl\") pod \"llmisvc-controller-manager-7f65c45d5c-7t766\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") " pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.467652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.467637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzltf\" (UniqueName: \"kubernetes.io/projected/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-kube-api-access-fzltf\") pod \"kserve-controller-manager-68bc99fcb8-2k89z\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") " pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:52.573101 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.573070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:52.701302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.701263    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7f65c45d5c-7t766"]
Jul 06 20:40:52.703504 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:40:52.703477    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2040c113_44c2_40c4_acf0_0cc9749c489a.slice/crio-19b3905fdb40f8b982636f3201bbd8ca4d03667b28d23a8c48f539c0ce723497 WatchSource:0}: Error finding container 19b3905fdb40f8b982636f3201bbd8ca4d03667b28d23a8c48f539c0ce723497: Status 404 returned error can't find the container with id 19b3905fdb40f8b982636f3201bbd8ca4d03667b28d23a8c48f539c0ce723497
Jul 06 20:40:52.704820 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.704801    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:40:52.843646 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.843607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766" event={"ID":"2040c113-44c2-40c4-acf0-0cc9749c489a","Type":"ContainerStarted","Data":"19b3905fdb40f8b982636f3201bbd8ca4d03667b28d23a8c48f539c0ce723497"}
Jul 06 20:40:52.956549 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.956462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert\") pod \"kserve-controller-manager-68bc99fcb8-2k89z\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") " pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:52.958817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:52.958787    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert\") pod \"kserve-controller-manager-68bc99fcb8-2k89z\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") " pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:53.147495 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:53.147460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:53.283985 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:53.283944    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-2k89z"]
Jul 06 20:40:53.285376 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:40:53.285352    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712eeea6_45b6_4fd4_bdcd_9a12b87d01fe.slice/crio-9bdefa9daa60c9a8162a0bcf8aa583f095280eff040cfcc052edfdeb958b5df5 WatchSource:0}: Error finding container 9bdefa9daa60c9a8162a0bcf8aa583f095280eff040cfcc052edfdeb958b5df5: Status 404 returned error can't find the container with id 9bdefa9daa60c9a8162a0bcf8aa583f095280eff040cfcc052edfdeb958b5df5
Jul 06 20:40:53.498901 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:53.498829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:53.595271 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:53.595237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:40:53.850359 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:53.850277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z" event={"ID":"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe","Type":"ContainerStarted","Data":"9bdefa9daa60c9a8162a0bcf8aa583f095280eff040cfcc052edfdeb958b5df5"}
Jul 06 20:40:56.865321 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:56.865212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766" event={"ID":"2040c113-44c2-40c4-acf0-0cc9749c489a","Type":"ContainerStarted","Data":"d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49"}
Jul 06 20:40:56.865750 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:56.865320    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:40:56.866795 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:56.866771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z" event={"ID":"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe","Type":"ContainerStarted","Data":"b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6"}
Jul 06 20:40:56.866933 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:56.866901    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:40:56.884042 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:56.883991    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766" podStartSLOduration=1.028026795 podStartE2EDuration="4.883974272s" totalImagesPullingTime="3.855947477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:56.882042543 +0000 UTC m=+354.216383142" watchObservedRunningTime="2026-07-06 20:40:56.883974272 +0000 UTC m=+354.218314859"
Jul 06 20:40:56.900901 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:40:56.900844    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z" podStartSLOduration=1.6269998810000001 podStartE2EDuration="4.900828793s" totalImagesPullingTime="3.273828912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:56.89916356 +0000 UTC m=+354.233504177" watchObservedRunningTime="2026-07-06 20:40:56.900828793 +0000 UTC m=+354.235169388"
Jul 06 20:41:27.873214 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:27.873182    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 20:41:27.876047 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:27.876026    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:41:29.324941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.324877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:29.426173 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.426137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:29.587403 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.587319    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"]
Jul 06 20:41:29.590895 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.590874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.593826 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.593786    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 06 20:41:29.596066 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.596041    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r744q\""
Jul 06 20:41:29.596159 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.596095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 06 20:41:29.599917 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.599897    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"]
Jul 06 20:41:29.673670 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.673640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.673886 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.673680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.673886 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.673703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx87f\" (UniqueName: \"kubernetes.io/projected/3cb138fb-de95-4bae-b79f-e62cd513bb1f-kube-api-access-hx87f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.774190 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.774157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.774344 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.774199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.774344 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.774224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx87f\" (UniqueName: \"kubernetes.io/projected/3cb138fb-de95-4bae-b79f-e62cd513bb1f-kube-api-access-hx87f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.774574 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.774555    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.774610 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.774570    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.786209 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.786185    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx87f\" (UniqueName: \"kubernetes.io/projected/3cb138fb-de95-4bae-b79f-e62cd513bb1f-kube-api-access-hx87f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:29.901294 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:29.901209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:30.034345 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:30.034316    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"]
Jul 06 20:41:30.037090 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:41:30.037067    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb138fb_de95_4bae_b79f_e62cd513bb1f.slice/crio-63dd7eafa004c02b64207aebbe42193e4632541c9bf755f4eed01c3d3ec981e8 WatchSource:0}: Error finding container 63dd7eafa004c02b64207aebbe42193e4632541c9bf755f4eed01c3d3ec981e8: Status 404 returned error can't find the container with id 63dd7eafa004c02b64207aebbe42193e4632541c9bf755f4eed01c3d3ec981e8
Jul 06 20:41:31.003881 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:31.003845    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerID="a357ca520d8cf442a311f8169699ade835b4e2e50c841a4b96ecd0e81789d96d" exitCode=0
Jul 06 20:41:31.004268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:31.003933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk" event={"ID":"3cb138fb-de95-4bae-b79f-e62cd513bb1f","Type":"ContainerDied","Data":"a357ca520d8cf442a311f8169699ade835b4e2e50c841a4b96ecd0e81789d96d"}
Jul 06 20:41:31.004268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:31.003995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk" event={"ID":"3cb138fb-de95-4bae-b79f-e62cd513bb1f","Type":"ContainerStarted","Data":"63dd7eafa004c02b64207aebbe42193e4632541c9bf755f4eed01c3d3ec981e8"}
Jul 06 20:41:41.048109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:41.048065    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerID="02a88324cec27ea424bedc9691ed504c7e60e35b5891a4726e5ed635333ed9ee" exitCode=0
Jul 06 20:41:41.048480 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:41.048146    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk" event={"ID":"3cb138fb-de95-4bae-b79f-e62cd513bb1f","Type":"ContainerDied","Data":"02a88324cec27ea424bedc9691ed504c7e60e35b5891a4726e5ed635333ed9ee"}
Jul 06 20:41:42.055172 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:42.055136    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerID="1fe3b8389d5e4e4fd561b50f467bc0f1082789fb92d5c9cbe502cef064e3ad69" exitCode=0
Jul 06 20:41:42.055518 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:42.055216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk" event={"ID":"3cb138fb-de95-4bae-b79f-e62cd513bb1f","Type":"ContainerDied","Data":"1fe3b8389d5e4e4fd561b50f467bc0f1082789fb92d5c9cbe502cef064e3ad69"}
Jul 06 20:41:43.184986 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.184942    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:43.288775 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.288741    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx87f\" (UniqueName: \"kubernetes.io/projected/3cb138fb-de95-4bae-b79f-e62cd513bb1f-kube-api-access-hx87f\") pod \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") "
Jul 06 20:41:43.288989 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.288792    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-util\") pod \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") "
Jul 06 20:41:43.288989 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.288836    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-bundle\") pod \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\" (UID: \"3cb138fb-de95-4bae-b79f-e62cd513bb1f\") "
Jul 06 20:41:43.289651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.289624    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-bundle" (OuterVolumeSpecName: "bundle") pod "3cb138fb-de95-4bae-b79f-e62cd513bb1f" (UID: "3cb138fb-de95-4bae-b79f-e62cd513bb1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:41:43.290904 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.290876    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb138fb-de95-4bae-b79f-e62cd513bb1f-kube-api-access-hx87f" (OuterVolumeSpecName: "kube-api-access-hx87f") pod "3cb138fb-de95-4bae-b79f-e62cd513bb1f" (UID: "3cb138fb-de95-4bae-b79f-e62cd513bb1f"). InnerVolumeSpecName "kube-api-access-hx87f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:41:43.296229 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.296203    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-util" (OuterVolumeSpecName: "util") pod "3cb138fb-de95-4bae-b79f-e62cd513bb1f" (UID: "3cb138fb-de95-4bae-b79f-e62cd513bb1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:41:43.389407 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.389371    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:41:43.389407 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.389403    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx87f\" (UniqueName: \"kubernetes.io/projected/3cb138fb-de95-4bae-b79f-e62cd513bb1f-kube-api-access-hx87f\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:41:43.389407 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:43.389412    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb138fb-de95-4bae-b79f-e62cd513bb1f-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:41:44.064936 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:44.064907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk" event={"ID":"3cb138fb-de95-4bae-b79f-e62cd513bb1f","Type":"ContainerDied","Data":"63dd7eafa004c02b64207aebbe42193e4632541c9bf755f4eed01c3d3ec981e8"}
Jul 06 20:41:44.065152 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:44.064948    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63dd7eafa004c02b64207aebbe42193e4632541c9bf755f4eed01c3d3ec981e8"
Jul 06 20:41:44.065152 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:44.064951    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407ngnk"
Jul 06 20:41:45.626041 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:45.625993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:46.357920 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:46.357889    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:49.470295 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:49.470251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:49.578025 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:49.577989    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:54.083740 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:54.083699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:54.143685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:54.143650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:57.013873 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:57.013842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:41:57.081406 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:41:57.081375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:07.487711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:07.487678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:07.588650 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:07.588613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:07.689069 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:07.689034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:09.795067 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:09.795035    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:09.885125 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:09.885088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:11.787394 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:11.787361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:11.967228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:11.967190    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:12.067455 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.067377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:12.167694 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.167661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:12.269391 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.269357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:12.909708 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.909672    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"]
Jul 06 20:42:12.910235 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.910216    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerName="pull"
Jul 06 20:42:12.910320 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.910238    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerName="pull"
Jul 06 20:42:12.910320 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.910257    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerName="util"
Jul 06 20:42:12.910320 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.910265    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerName="util"
Jul 06 20:42:12.910320 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.910292    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerName="extract"
Jul 06 20:42:12.910320 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.910301    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerName="extract"
Jul 06 20:42:12.910562 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.910380    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb138fb-de95-4bae-b79f-e62cd513bb1f" containerName="extract"
Jul 06 20:42:12.914770 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.914749    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:12.917628 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.917607    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r744q\""
Jul 06 20:42:12.917731 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.917617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 06 20:42:12.918777 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.918762    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 06 20:42:12.927132 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:12.927111    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"]
Jul 06 20:42:13.051873 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.051829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.052087 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.051990    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.052087 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.052027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qj47\" (UniqueName: \"kubernetes.io/projected/ba1a3bbc-2add-4006-9a2e-48df395351f3-kube-api-access-2qj47\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.152719 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.152679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.152907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.152776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.152907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.152802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qj47\" (UniqueName: \"kubernetes.io/projected/ba1a3bbc-2add-4006-9a2e-48df395351f3-kube-api-access-2qj47\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.153082 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.153060    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.153145 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.153110    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.161569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.161514    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qj47\" (UniqueName: \"kubernetes.io/projected/ba1a3bbc-2add-4006-9a2e-48df395351f3-kube-api-access-2qj47\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.223819 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.223784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:13.359930 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:13.359908    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"]
Jul 06 20:42:13.362391 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:42:13.362363    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1a3bbc_2add_4006_9a2e_48df395351f3.slice/crio-e9a461d0a573a4e8dcb5cd822cc78272bda02940e5d4db10131dbe915ba07bef WatchSource:0}: Error finding container e9a461d0a573a4e8dcb5cd822cc78272bda02940e5d4db10131dbe915ba07bef: Status 404 returned error can't find the container with id e9a461d0a573a4e8dcb5cd822cc78272bda02940e5d4db10131dbe915ba07bef
Jul 06 20:42:14.195695 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:14.195662    2582 generic.go:356] "Generic (PLEG): container finished" podID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerID="7f08cdc0b726e85a7f4c349a3a088750b17e328696b0eba59c25387948a21282" exitCode=0
Jul 06 20:42:14.196094 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:14.195753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9" event={"ID":"ba1a3bbc-2add-4006-9a2e-48df395351f3","Type":"ContainerDied","Data":"7f08cdc0b726e85a7f4c349a3a088750b17e328696b0eba59c25387948a21282"}
Jul 06 20:42:14.196094 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:14.195817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9" event={"ID":"ba1a3bbc-2add-4006-9a2e-48df395351f3","Type":"ContainerStarted","Data":"e9a461d0a573a4e8dcb5cd822cc78272bda02940e5d4db10131dbe915ba07bef"}
Jul 06 20:42:17.211355 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:17.211316    2582 generic.go:356] "Generic (PLEG): container finished" podID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerID="fa1892b6ba38f1fe4f224ce190f728bc835eb5253c9d5cd2c22c79100a5d5e57" exitCode=0
Jul 06 20:42:17.211355 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:17.211357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9" event={"ID":"ba1a3bbc-2add-4006-9a2e-48df395351f3","Type":"ContainerDied","Data":"fa1892b6ba38f1fe4f224ce190f728bc835eb5253c9d5cd2c22c79100a5d5e57"}
Jul 06 20:42:17.264568 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:17.264546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:17.326534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:17.326501    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:18.217452 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:18.217414    2582 generic.go:356] "Generic (PLEG): container finished" podID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerID="5788a1af84d0eabcf589e1a9a497c6ea045abb67d23e36f48d37941b488e509e" exitCode=0
Jul 06 20:42:18.217843 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:18.217477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9" event={"ID":"ba1a3bbc-2add-4006-9a2e-48df395351f3","Type":"ContainerDied","Data":"5788a1af84d0eabcf589e1a9a497c6ea045abb67d23e36f48d37941b488e509e"}
Jul 06 20:42:19.351218 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.351193    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:19.404948 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.404921    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-util\") pod \"ba1a3bbc-2add-4006-9a2e-48df395351f3\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") "
Jul 06 20:42:19.405111 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.404956    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-bundle\") pod \"ba1a3bbc-2add-4006-9a2e-48df395351f3\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") "
Jul 06 20:42:19.405111 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.405027    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qj47\" (UniqueName: \"kubernetes.io/projected/ba1a3bbc-2add-4006-9a2e-48df395351f3-kube-api-access-2qj47\") pod \"ba1a3bbc-2add-4006-9a2e-48df395351f3\" (UID: \"ba1a3bbc-2add-4006-9a2e-48df395351f3\") "
Jul 06 20:42:19.405386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.405347    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-bundle" (OuterVolumeSpecName: "bundle") pod "ba1a3bbc-2add-4006-9a2e-48df395351f3" (UID: "ba1a3bbc-2add-4006-9a2e-48df395351f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:42:19.407139 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.407114    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1a3bbc-2add-4006-9a2e-48df395351f3-kube-api-access-2qj47" (OuterVolumeSpecName: "kube-api-access-2qj47") pod "ba1a3bbc-2add-4006-9a2e-48df395351f3" (UID: "ba1a3bbc-2add-4006-9a2e-48df395351f3"). InnerVolumeSpecName "kube-api-access-2qj47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:42:19.410359 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.410340    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-util" (OuterVolumeSpecName: "util") pod "ba1a3bbc-2add-4006-9a2e-48df395351f3" (UID: "ba1a3bbc-2add-4006-9a2e-48df395351f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:42:19.505874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.505786    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:19.505874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.505820    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba1a3bbc-2add-4006-9a2e-48df395351f3-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:19.505874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:19.505829    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qj47\" (UniqueName: \"kubernetes.io/projected/ba1a3bbc-2add-4006-9a2e-48df395351f3-kube-api-access-2qj47\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:20.227204 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:20.227173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9" event={"ID":"ba1a3bbc-2add-4006-9a2e-48df395351f3","Type":"ContainerDied","Data":"e9a461d0a573a4e8dcb5cd822cc78272bda02940e5d4db10131dbe915ba07bef"}
Jul 06 20:42:20.227204 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:20.227190    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8fwr9"
Jul 06 20:42:20.227399 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:20.227219    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9a461d0a573a4e8dcb5cd822cc78272bda02940e5d4db10131dbe915ba07bef"
Jul 06 20:42:20.582295 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:20.582181    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:25.223012 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:25.222911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:25.312703 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:25.312664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:35.979920 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:35.979890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:41.136868 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:41.136836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:41.214594 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:41.214563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:42.561139 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:42.561098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:44.828496 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:44.828468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:44.927478 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:44.927446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:47.240289 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240258    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"]
Jul 06 20:42:47.240684 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240670    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerName="extract"
Jul 06 20:42:47.240724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240687    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerName="extract"
Jul 06 20:42:47.240724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240698    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerName="util"
Jul 06 20:42:47.240724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240703    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerName="util"
Jul 06 20:42:47.240724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240709    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerName="pull"
Jul 06 20:42:47.240724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerName="pull"
Jul 06 20:42:47.240870 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.240778    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1a3bbc-2add-4006-9a2e-48df395351f3" containerName="extract"
Jul 06 20:42:47.245374 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.245355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.248809 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.248785    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 20:42:47.248809 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.248799    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-w7p6s\""
Jul 06 20:42:47.261237 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.261208    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"]
Jul 06 20:42:47.263788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.263763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:47.326604 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.326577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:47.341014 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.340987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-credential-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341140 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341140 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341140 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-data\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341245 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqlxq\" (UniqueName: \"kubernetes.io/projected/dfe3aa62-0935-48f7-a93d-8d21917d35a5-kube-api-access-mqlxq\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-token\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341311 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-workload-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341311 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-workload-certs\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.341379 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.341334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442010 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.441951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-workload-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442194 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-workload-certs\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442194 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442427 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442402    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-workload-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442487 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-credential-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442487 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442415    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-workload-certs\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442811 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442926 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442785    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-credential-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.442926 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.443087 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.442928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-data\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.443147 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.443077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqlxq\" (UniqueName: \"kubernetes.io/projected/dfe3aa62-0935-48f7-a93d-8d21917d35a5-kube-api-access-mqlxq\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.443147 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.443126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-token\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.443250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.443164    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.443303 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.443239    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-data\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.444999 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.444976    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.445333 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.445312    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.452409 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.452384    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe3aa62-0935-48f7-a93d-8d21917d35a5-istio-token\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.452579 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.452559    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqlxq\" (UniqueName: \"kubernetes.io/projected/dfe3aa62-0935-48f7-a93d-8d21917d35a5-kube-api-access-mqlxq\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-vmnww\" (UID: \"dfe3aa62-0935-48f7-a93d-8d21917d35a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.555577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.555472    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:47.697920 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:47.697888    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"]
Jul 06 20:42:47.699542 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:42:47.699513    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe3aa62_0935_48f7_a93d_8d21917d35a5.slice/crio-9f808bb8f669aaa68b7b11ff4c98725a8bbc03632f0fa45b93d6d63013b3df87 WatchSource:0}: Error finding container 9f808bb8f669aaa68b7b11ff4c98725a8bbc03632f0fa45b93d6d63013b3df87: Status 404 returned error can't find the container with id 9f808bb8f669aaa68b7b11ff4c98725a8bbc03632f0fa45b93d6d63013b3df87
Jul 06 20:42:48.349814 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:48.349781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"9f808bb8f669aaa68b7b11ff4c98725a8bbc03632f0fa45b93d6d63013b3df87"}
Jul 06 20:42:50.299457 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:50.299409    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:42:50.299717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:50.299499    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:42:50.299717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:50.299528    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:42:50.698163 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:50.698119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:51.368830 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:51.368794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"b7e7f760ad637870ff4c58e43d1480f88326c91579dfa567ba3d521fb9f7ddf2"}
Jul 06 20:42:51.555617 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:51.555588    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:51.560318 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:51.560292    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:51.595548 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:51.595499    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podStartSLOduration=1.9987532209999999 podStartE2EDuration="4.59548529s" totalImagesPullingTime="2.596732069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:51.400380284 +0000 UTC m=+468.734720879" watchObservedRunningTime="2026-07-06 20:42:51.59548529 +0000 UTC m=+468.929825888"
Jul 06 20:42:52.373796 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:52.373755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:52.374798 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:52.374782    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:42:52.824861 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:52.824824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:52.925365 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:52.925337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:42:54.904059 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:54.904022    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"]
Jul 06 20:42:54.907551 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:54.907533    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:54.910470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:54.910451    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 06 20:42:54.910470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:54.910466    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 06 20:42:54.910633 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:54.910451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r744q\""
Jul 06 20:42:54.916830 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:54.916810    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"]
Jul 06 20:42:54.997539 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:54.997506    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"]
Jul 06 20:42:55.000939 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.000922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.009861 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.009834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gk6\" (UniqueName: \"kubernetes.io/projected/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-kube-api-access-b4gk6\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.010015 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.009896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.010015 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.009935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.010262 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.010244    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"]
Jul 06 20:42:55.095878 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.095841    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"]
Jul 06 20:42:55.099333 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.099310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.108397 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.108376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"]
Jul 06 20:42:55.110579 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.110557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.110694 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.110586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.110694 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.110608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gtb\" (UniqueName: \"kubernetes.io/projected/239c43e4-aeda-4635-a651-81b1877b7974-kube-api-access-56gtb\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.110808 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.110733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gk6\" (UniqueName: \"kubernetes.io/projected/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-kube-api-access-b4gk6\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.110871 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.110813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.110871 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.110854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.111010 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.110997    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.111175 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.111152    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.119868 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.119842    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gk6\" (UniqueName: \"kubernetes.io/projected/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-kube-api-access-b4gk6\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.201295 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.201260    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"]
Jul 06 20:42:55.204869 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.204852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.211391 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.211369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gtb\" (UniqueName: \"kubernetes.io/projected/239c43e4-aeda-4635-a651-81b1877b7974-kube-api-access-56gtb\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.211476 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.211404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.211476 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.211450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.211588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.211477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhc6v\" (UniqueName: \"kubernetes.io/projected/0a95a42c-40db-448c-ba5b-72067fe915ce-kube-api-access-hhc6v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.211754 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.211618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.211897 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.211877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.212544 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.212485    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.212544 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.212523    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.214807 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.214787    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"]
Jul 06 20:42:55.217712 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.217692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:42:55.220553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.220532    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gtb\" (UniqueName: \"kubernetes.io/projected/239c43e4-aeda-4635-a651-81b1877b7974-kube-api-access-56gtb\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.310287 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.310247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:42:55.312728 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.312699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c82f\" (UniqueName: \"kubernetes.io/projected/1664df93-fbaa-40bf-8c1d-a33fdb337701-kube-api-access-4c82f\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.312874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.312767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.312874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.312831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.312992 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.312886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.312992 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.312956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc6v\" (UniqueName: \"kubernetes.io/projected/0a95a42c-40db-448c-ba5b-72067fe915ce-kube-api-access-hhc6v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.313111 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.313009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.313194 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.313165    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.313241 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.313213    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.322857 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.322829    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc6v\" (UniqueName: \"kubernetes.io/projected/0a95a42c-40db-448c-ba5b-72067fe915ce-kube-api-access-hhc6v\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.354342 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.354318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"]
Jul 06 20:42:55.357244 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:42:55.357218    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dcaa82_8266_45bb_a6f3_3fefa08cebf9.slice/crio-0577e09dff5da1476ce54bdf2ae686e3d7cb7a9fc7ea408fa26c41d0ce7c85d9 WatchSource:0}: Error finding container 0577e09dff5da1476ce54bdf2ae686e3d7cb7a9fc7ea408fa26c41d0ce7c85d9: Status 404 returned error can't find the container with id 0577e09dff5da1476ce54bdf2ae686e3d7cb7a9fc7ea408fa26c41d0ce7c85d9
Jul 06 20:42:55.388032 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.388002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv" event={"ID":"42dcaa82-8266-45bb-a6f3-3fefa08cebf9","Type":"ContainerStarted","Data":"0577e09dff5da1476ce54bdf2ae686e3d7cb7a9fc7ea408fa26c41d0ce7c85d9"}
Jul 06 20:42:55.409154 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.409130    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:55.414040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.413941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.414040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.414024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.414234 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.414113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c82f\" (UniqueName: \"kubernetes.io/projected/1664df93-fbaa-40bf-8c1d-a33fdb337701-kube-api-access-4c82f\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.414613 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.414591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.414764 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.414688    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.423252 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.423201    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c82f\" (UniqueName: \"kubernetes.io/projected/1664df93-fbaa-40bf-8c1d-a33fdb337701-kube-api-access-4c82f\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.459014 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.458977    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"]
Jul 06 20:42:55.459855 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:42:55.459826    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239c43e4_aeda_4635_a651_81b1877b7974.slice/crio-d042a389f7e4b5e3b4c800a55cafc46552b31bffbb5d5ba0b368d114e1799147 WatchSource:0}: Error finding container d042a389f7e4b5e3b4c800a55cafc46552b31bffbb5d5ba0b368d114e1799147: Status 404 returned error can't find the container with id d042a389f7e4b5e3b4c800a55cafc46552b31bffbb5d5ba0b368d114e1799147
Jul 06 20:42:55.514879 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.514850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:55.559322 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.559299    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"]
Jul 06 20:42:55.562474 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:42:55.562448    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a95a42c_40db_448c_ba5b_72067fe915ce.slice/crio-5e75862f203a622c0c219dbba5e0bd583ec3123a18d1fc24431be925693c24f9 WatchSource:0}: Error finding container 5e75862f203a622c0c219dbba5e0bd583ec3123a18d1fc24431be925693c24f9: Status 404 returned error can't find the container with id 5e75862f203a622c0c219dbba5e0bd583ec3123a18d1fc24431be925693c24f9
Jul 06 20:42:55.662109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:55.662077    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"]
Jul 06 20:42:55.665878 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:42:55.665849    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1664df93_fbaa_40bf_8c1d_a33fdb337701.slice/crio-137addd591e201d85f60d86878f33cfec3fd6e33fa466127d814623660f61e52 WatchSource:0}: Error finding container 137addd591e201d85f60d86878f33cfec3fd6e33fa466127d814623660f61e52: Status 404 returned error can't find the container with id 137addd591e201d85f60d86878f33cfec3fd6e33fa466127d814623660f61e52
Jul 06 20:42:56.399242 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.399203    2582 generic.go:356] "Generic (PLEG): container finished" podID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerID="b61109ff38f2bf4bc575dce1a48658128b0d9661b124bde53d922c9737ae4ac0" exitCode=0
Jul 06 20:42:56.399711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.399251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv" event={"ID":"42dcaa82-8266-45bb-a6f3-3fefa08cebf9","Type":"ContainerDied","Data":"b61109ff38f2bf4bc575dce1a48658128b0d9661b124bde53d922c9737ae4ac0"}
Jul 06 20:42:56.400870 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.400855    2582 generic.go:356] "Generic (PLEG): container finished" podID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerID="56bf44e0ba0b96e8c883e7f751c5d68a5ae01ae230e5ff62096f6456077e1884" exitCode=0
Jul 06 20:42:56.400931 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.400917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x" event={"ID":"1664df93-fbaa-40bf-8c1d-a33fdb337701","Type":"ContainerDied","Data":"56bf44e0ba0b96e8c883e7f751c5d68a5ae01ae230e5ff62096f6456077e1884"}
Jul 06 20:42:56.401002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.400956    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x" event={"ID":"1664df93-fbaa-40bf-8c1d-a33fdb337701","Type":"ContainerStarted","Data":"137addd591e201d85f60d86878f33cfec3fd6e33fa466127d814623660f61e52"}
Jul 06 20:42:56.402468 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.402421    2582 generic.go:356] "Generic (PLEG): container finished" podID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerID="c111b0c13a48c01a4a2a6c45a6759fd514d1b09d604a2522591c348cd3324eeb" exitCode=0
Jul 06 20:42:56.402537 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.402505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z" event={"ID":"0a95a42c-40db-448c-ba5b-72067fe915ce","Type":"ContainerDied","Data":"c111b0c13a48c01a4a2a6c45a6759fd514d1b09d604a2522591c348cd3324eeb"}
Jul 06 20:42:56.402595 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.402567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z" event={"ID":"0a95a42c-40db-448c-ba5b-72067fe915ce","Type":"ContainerStarted","Data":"5e75862f203a622c0c219dbba5e0bd583ec3123a18d1fc24431be925693c24f9"}
Jul 06 20:42:56.404171 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.404153    2582 generic.go:356] "Generic (PLEG): container finished" podID="239c43e4-aeda-4635-a651-81b1877b7974" containerID="f5f030b301c90051bb5944ad0e02750cb9bd84459b20670771b2caa217052944" exitCode=0
Jul 06 20:42:56.404250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.404208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz" event={"ID":"239c43e4-aeda-4635-a651-81b1877b7974","Type":"ContainerDied","Data":"f5f030b301c90051bb5944ad0e02750cb9bd84459b20670771b2caa217052944"}
Jul 06 20:42:56.404250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:56.404238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz" event={"ID":"239c43e4-aeda-4635-a651-81b1877b7974","Type":"ContainerStarted","Data":"d042a389f7e4b5e3b4c800a55cafc46552b31bffbb5d5ba0b368d114e1799147"}
Jul 06 20:42:57.413676 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:57.413643    2582 generic.go:356] "Generic (PLEG): container finished" podID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerID="36e91868ab8dcff42cf5cae155c76553a251105e1d804a352d030e57c5132725" exitCode=0
Jul 06 20:42:57.414109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:57.413740    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x" event={"ID":"1664df93-fbaa-40bf-8c1d-a33fdb337701","Type":"ContainerDied","Data":"36e91868ab8dcff42cf5cae155c76553a251105e1d804a352d030e57c5132725"}
Jul 06 20:42:57.415606 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:57.415579    2582 generic.go:356] "Generic (PLEG): container finished" podID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerID="4b755e835480d0b774c264c4edbc7ad0e6cfe5df955f8ec5d5cf59078fa3690e" exitCode=0
Jul 06 20:42:57.415702 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:57.415652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z" event={"ID":"0a95a42c-40db-448c-ba5b-72067fe915ce","Type":"ContainerDied","Data":"4b755e835480d0b774c264c4edbc7ad0e6cfe5df955f8ec5d5cf59078fa3690e"}
Jul 06 20:42:58.422882 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.422800    2582 generic.go:356] "Generic (PLEG): container finished" podID="239c43e4-aeda-4635-a651-81b1877b7974" containerID="356ef5f6c4c1bd6f3266edcf3a87a6bac421f3f6e8a88896701709669933e60f" exitCode=0
Jul 06 20:42:58.423401 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.422895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz" event={"ID":"239c43e4-aeda-4635-a651-81b1877b7974","Type":"ContainerDied","Data":"356ef5f6c4c1bd6f3266edcf3a87a6bac421f3f6e8a88896701709669933e60f"}
Jul 06 20:42:58.424749 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.424726    2582 generic.go:356] "Generic (PLEG): container finished" podID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerID="821943f686681ae71344b0f711d2ccbcd943aef25b74c6b3f075160d2d2586f8" exitCode=0
Jul 06 20:42:58.424821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.424804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv" event={"ID":"42dcaa82-8266-45bb-a6f3-3fefa08cebf9","Type":"ContainerDied","Data":"821943f686681ae71344b0f711d2ccbcd943aef25b74c6b3f075160d2d2586f8"}
Jul 06 20:42:58.426878 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.426857    2582 generic.go:356] "Generic (PLEG): container finished" podID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerID="17b6a3a2c113d671cebd5526674e91a46812caed90410fb95307377a1cecf7b2" exitCode=0
Jul 06 20:42:58.426976 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.426923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x" event={"ID":"1664df93-fbaa-40bf-8c1d-a33fdb337701","Type":"ContainerDied","Data":"17b6a3a2c113d671cebd5526674e91a46812caed90410fb95307377a1cecf7b2"}
Jul 06 20:42:58.429038 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.429014    2582 generic.go:356] "Generic (PLEG): container finished" podID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerID="0ddf762f9727f87a5172d11fd946d4bb8d86f776373a5fc33ed070091c5d622e" exitCode=0
Jul 06 20:42:58.429123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:58.429093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z" event={"ID":"0a95a42c-40db-448c-ba5b-72067fe915ce","Type":"ContainerDied","Data":"0ddf762f9727f87a5172d11fd946d4bb8d86f776373a5fc33ed070091c5d622e"}
Jul 06 20:42:59.435403 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.435374    2582 generic.go:356] "Generic (PLEG): container finished" podID="239c43e4-aeda-4635-a651-81b1877b7974" containerID="4757f729ebe91dfce548657e88fe39859c2e86c1f1021e9f3756f70976f2085b" exitCode=0
Jul 06 20:42:59.435842 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.435447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz" event={"ID":"239c43e4-aeda-4635-a651-81b1877b7974","Type":"ContainerDied","Data":"4757f729ebe91dfce548657e88fe39859c2e86c1f1021e9f3756f70976f2085b"}
Jul 06 20:42:59.437560 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.437541    2582 generic.go:356] "Generic (PLEG): container finished" podID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerID="8bde88ad1f1dd138ead063b4c9f257c7b27a0ab8086fb3650117e0f5a1f64781" exitCode=0
Jul 06 20:42:59.437676 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.437626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv" event={"ID":"42dcaa82-8266-45bb-a6f3-3fefa08cebf9","Type":"ContainerDied","Data":"8bde88ad1f1dd138ead063b4c9f257c7b27a0ab8086fb3650117e0f5a1f64781"}
Jul 06 20:42:59.578403 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.578376    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:42:59.606014 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.605988    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:42:59.753232 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753131    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-bundle\") pod \"0a95a42c-40db-448c-ba5b-72067fe915ce\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") "
Jul 06 20:42:59.753232 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753196    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-bundle\") pod \"1664df93-fbaa-40bf-8c1d-a33fdb337701\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") "
Jul 06 20:42:59.753232 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c82f\" (UniqueName: \"kubernetes.io/projected/1664df93-fbaa-40bf-8c1d-a33fdb337701-kube-api-access-4c82f\") pod \"1664df93-fbaa-40bf-8c1d-a33fdb337701\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") "
Jul 06 20:42:59.753500 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753250    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-util\") pod \"1664df93-fbaa-40bf-8c1d-a33fdb337701\" (UID: \"1664df93-fbaa-40bf-8c1d-a33fdb337701\") "
Jul 06 20:42:59.753500 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhc6v\" (UniqueName: \"kubernetes.io/projected/0a95a42c-40db-448c-ba5b-72067fe915ce-kube-api-access-hhc6v\") pod \"0a95a42c-40db-448c-ba5b-72067fe915ce\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") "
Jul 06 20:42:59.753500 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753346    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-util\") pod \"0a95a42c-40db-448c-ba5b-72067fe915ce\" (UID: \"0a95a42c-40db-448c-ba5b-72067fe915ce\") "
Jul 06 20:42:59.753888 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753852    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-bundle" (OuterVolumeSpecName: "bundle") pod "0a95a42c-40db-448c-ba5b-72067fe915ce" (UID: "0a95a42c-40db-448c-ba5b-72067fe915ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:42:59.754016 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.753941    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-bundle" (OuterVolumeSpecName: "bundle") pod "1664df93-fbaa-40bf-8c1d-a33fdb337701" (UID: "1664df93-fbaa-40bf-8c1d-a33fdb337701"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:42:59.755455 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.755430    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1664df93-fbaa-40bf-8c1d-a33fdb337701-kube-api-access-4c82f" (OuterVolumeSpecName: "kube-api-access-4c82f") pod "1664df93-fbaa-40bf-8c1d-a33fdb337701" (UID: "1664df93-fbaa-40bf-8c1d-a33fdb337701"). InnerVolumeSpecName "kube-api-access-4c82f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:42:59.755574 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.755552    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a95a42c-40db-448c-ba5b-72067fe915ce-kube-api-access-hhc6v" (OuterVolumeSpecName: "kube-api-access-hhc6v") pod "0a95a42c-40db-448c-ba5b-72067fe915ce" (UID: "0a95a42c-40db-448c-ba5b-72067fe915ce"). InnerVolumeSpecName "kube-api-access-hhc6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:42:59.761590 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.761563    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-util" (OuterVolumeSpecName: "util") pod "1664df93-fbaa-40bf-8c1d-a33fdb337701" (UID: "1664df93-fbaa-40bf-8c1d-a33fdb337701"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:42:59.762127 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.762106    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-util" (OuterVolumeSpecName: "util") pod "0a95a42c-40db-448c-ba5b-72067fe915ce" (UID: "0a95a42c-40db-448c-ba5b-72067fe915ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:42:59.854309 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.854260    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:59.854309 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.854303    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:59.854309 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.854313    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c82f\" (UniqueName: \"kubernetes.io/projected/1664df93-fbaa-40bf-8c1d-a33fdb337701-kube-api-access-4c82f\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:59.854512 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.854322    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1664df93-fbaa-40bf-8c1d-a33fdb337701-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:59.854512 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.854331    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhc6v\" (UniqueName: \"kubernetes.io/projected/0a95a42c-40db-448c-ba5b-72067fe915ce-kube-api-access-hhc6v\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:42:59.854512 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:42:59.854339    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a95a42c-40db-448c-ba5b-72067fe915ce-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:43:00.445352 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.445320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x"
Jul 06 20:43:00.445769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.445316    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabmvs7x" event={"ID":"1664df93-fbaa-40bf-8c1d-a33fdb337701","Type":"ContainerDied","Data":"137addd591e201d85f60d86878f33cfec3fd6e33fa466127d814623660f61e52"}
Jul 06 20:43:00.445769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.445444    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137addd591e201d85f60d86878f33cfec3fd6e33fa466127d814623660f61e52"
Jul 06 20:43:00.447281 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.447263    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z"
Jul 06 20:43:00.447396 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.447299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36tcn2z" event={"ID":"0a95a42c-40db-448c-ba5b-72067fe915ce","Type":"ContainerDied","Data":"5e75862f203a622c0c219dbba5e0bd583ec3123a18d1fc24431be925693c24f9"}
Jul 06 20:43:00.447396 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.447337    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e75862f203a622c0c219dbba5e0bd583ec3123a18d1fc24431be925693c24f9"
Jul 06 20:43:00.603314 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.603295    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:43:00.605675 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.605659    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:43:00.759863 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.759776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-util\") pod \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") "
Jul 06 20:43:00.760054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.759881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56gtb\" (UniqueName: \"kubernetes.io/projected/239c43e4-aeda-4635-a651-81b1877b7974-kube-api-access-56gtb\") pod \"239c43e4-aeda-4635-a651-81b1877b7974\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") "
Jul 06 20:43:00.760054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.759910    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-bundle\") pod \"239c43e4-aeda-4635-a651-81b1877b7974\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") "
Jul 06 20:43:00.760054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.759943    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-bundle\") pod \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") "
Jul 06 20:43:00.760054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.759990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-util\") pod \"239c43e4-aeda-4635-a651-81b1877b7974\" (UID: \"239c43e4-aeda-4635-a651-81b1877b7974\") "
Jul 06 20:43:00.760054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.760023    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4gk6\" (UniqueName: \"kubernetes.io/projected/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-kube-api-access-b4gk6\") pod \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\" (UID: \"42dcaa82-8266-45bb-a6f3-3fefa08cebf9\") "
Jul 06 20:43:00.760522 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.760443    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-bundle" (OuterVolumeSpecName: "bundle") pod "239c43e4-aeda-4635-a651-81b1877b7974" (UID: "239c43e4-aeda-4635-a651-81b1877b7974"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:43:00.760707 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.760679    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-bundle" (OuterVolumeSpecName: "bundle") pod "42dcaa82-8266-45bb-a6f3-3fefa08cebf9" (UID: "42dcaa82-8266-45bb-a6f3-3fefa08cebf9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:43:00.762233 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.762210    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-kube-api-access-b4gk6" (OuterVolumeSpecName: "kube-api-access-b4gk6") pod "42dcaa82-8266-45bb-a6f3-3fefa08cebf9" (UID: "42dcaa82-8266-45bb-a6f3-3fefa08cebf9"). InnerVolumeSpecName "kube-api-access-b4gk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:43:00.762404 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.762388    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239c43e4-aeda-4635-a651-81b1877b7974-kube-api-access-56gtb" (OuterVolumeSpecName: "kube-api-access-56gtb") pod "239c43e4-aeda-4635-a651-81b1877b7974" (UID: "239c43e4-aeda-4635-a651-81b1877b7974"). InnerVolumeSpecName "kube-api-access-56gtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:43:00.766515 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.766482    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-util" (OuterVolumeSpecName: "util") pod "239c43e4-aeda-4635-a651-81b1877b7974" (UID: "239c43e4-aeda-4635-a651-81b1877b7974"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:43:00.766710 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.766689    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-util" (OuterVolumeSpecName: "util") pod "42dcaa82-8266-45bb-a6f3-3fefa08cebf9" (UID: "42dcaa82-8266-45bb-a6f3-3fefa08cebf9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:43:00.860817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.860777    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gk6\" (UniqueName: \"kubernetes.io/projected/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-kube-api-access-b4gk6\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:43:00.860817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.860808    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:43:00.860817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.860817    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56gtb\" (UniqueName: \"kubernetes.io/projected/239c43e4-aeda-4635-a651-81b1877b7974-kube-api-access-56gtb\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:43:00.860817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.860828    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:43:00.861101 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.860837    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42dcaa82-8266-45bb-a6f3-3fefa08cebf9-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:43:00.861101 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:00.860844    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/239c43e4-aeda-4635-a651-81b1877b7974-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:43:01.453428 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:01.453398    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz"
Jul 06 20:43:01.453428 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:01.453413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efvkdzz" event={"ID":"239c43e4-aeda-4635-a651-81b1877b7974","Type":"ContainerDied","Data":"d042a389f7e4b5e3b4c800a55cafc46552b31bffbb5d5ba0b368d114e1799147"}
Jul 06 20:43:01.453872 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:01.453461    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d042a389f7e4b5e3b4c800a55cafc46552b31bffbb5d5ba0b368d114e1799147"
Jul 06 20:43:01.455402 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:01.455378    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv"
Jul 06 20:43:01.455594 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:01.455384    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76sm7zv" event={"ID":"42dcaa82-8266-45bb-a6f3-3fefa08cebf9","Type":"ContainerDied","Data":"0577e09dff5da1476ce54bdf2ae686e3d7cb7a9fc7ea408fa26c41d0ce7c85d9"}
Jul 06 20:43:01.455655 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:01.455615    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0577e09dff5da1476ce54bdf2ae686e3d7cb7a9fc7ea408fa26c41d0ce7c85d9"
Jul 06 20:43:03.747023 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:03.746975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:05.909690 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.909657    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"]
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910008    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerName="pull"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerName="pull"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910029    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239c43e4-aeda-4635-a651-81b1877b7974" containerName="pull"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910034    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="239c43e4-aeda-4635-a651-81b1877b7974" containerName="pull"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910040    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerName="pull"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910045    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerName="pull"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910051    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239c43e4-aeda-4635-a651-81b1877b7974" containerName="extract"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910057    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="239c43e4-aeda-4635-a651-81b1877b7974" containerName="extract"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910065    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerName="util"
Jul 06 20:43:05.910065 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910071    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerName="util"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910079    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239c43e4-aeda-4635-a651-81b1877b7974" containerName="util"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910084    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="239c43e4-aeda-4635-a651-81b1877b7974" containerName="util"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910090    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerName="util"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910095    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerName="util"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910104    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerName="util"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910108    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerName="util"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910116    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910121    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910127    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910132    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910138    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerName="pull"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910142    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerName="pull"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910147    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910152    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910200    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a95a42c-40db-448c-ba5b-72067fe915ce" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910207    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42dcaa82-8266-45bb-a6f3-3fefa08cebf9" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910216    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1664df93-fbaa-40bf-8c1d-a33fdb337701" containerName="extract"
Jul 06 20:43:05.910353 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.910222    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="239c43e4-aeda-4635-a651-81b1877b7974" containerName="extract"
Jul 06 20:43:05.915921 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.915906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"
Jul 06 20:43:05.919875 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.919853    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 20:43:05.921048 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.921025    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-crrdr\""
Jul 06 20:43:05.921151 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.921029    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 20:43:05.935279 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:05.935255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"]
Jul 06 20:43:06.008284 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.008245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxgw\" (UniqueName: \"kubernetes.io/projected/e8884f9b-a492-4efa-91c2-a79975e1b0ea-kube-api-access-5rxgw\") pod \"limitador-operator-controller-manager-6f9f468797-s2frj\" (UID: \"e8884f9b-a492-4efa-91c2-a79975e1b0ea\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"
Jul 06 20:43:06.109707 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.109672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxgw\" (UniqueName: \"kubernetes.io/projected/e8884f9b-a492-4efa-91c2-a79975e1b0ea-kube-api-access-5rxgw\") pod \"limitador-operator-controller-manager-6f9f468797-s2frj\" (UID: \"e8884f9b-a492-4efa-91c2-a79975e1b0ea\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"
Jul 06 20:43:06.124993 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.124934    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxgw\" (UniqueName: \"kubernetes.io/projected/e8884f9b-a492-4efa-91c2-a79975e1b0ea-kube-api-access-5rxgw\") pod \"limitador-operator-controller-manager-6f9f468797-s2frj\" (UID: \"e8884f9b-a492-4efa-91c2-a79975e1b0ea\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"
Jul 06 20:43:06.188827 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.188795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:06.225768 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.225742    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"
Jul 06 20:43:06.371230 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.371204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"]
Jul 06 20:43:06.371931 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:43:06.371909    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8884f9b_a492_4efa_91c2_a79975e1b0ea.slice/crio-9f5798aa1730d59f08aabb22a8bca76d2085a04c23f31acaa630015719bcc76e WatchSource:0}: Error finding container 9f5798aa1730d59f08aabb22a8bca76d2085a04c23f31acaa630015719bcc76e: Status 404 returned error can't find the container with id 9f5798aa1730d59f08aabb22a8bca76d2085a04c23f31acaa630015719bcc76e
Jul 06 20:43:06.480430 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.480331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj" event={"ID":"e8884f9b-a492-4efa-91c2-a79975e1b0ea","Type":"ContainerStarted","Data":"9f5798aa1730d59f08aabb22a8bca76d2085a04c23f31acaa630015719bcc76e"}
Jul 06 20:43:06.480656 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.480574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:06.661495 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.661461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:06.846460 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:06.846364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:08.491416 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.491388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj" event={"ID":"e8884f9b-a492-4efa-91c2-a79975e1b0ea","Type":"ContainerStarted","Data":"2939cbb243a4a82e68e022ba28ef82433431fa49ffff8018546943c4071d09fe"}
Jul 06 20:43:08.491740 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.491443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"
Jul 06 20:43:08.549009 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.548949    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj" podStartSLOduration=1.514369407 podStartE2EDuration="3.548932744s" totalImagesPullingTime="2.034563337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:08.546069138 +0000 UTC m=+485.880409734" watchObservedRunningTime="2026-07-06 20:43:08.548932744 +0000 UTC m=+485.883273340"
Jul 06 20:43:08.598863 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.598823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:08.931672 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.931638    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-gmblh"]
Jul 06 20:43:08.936051 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.936035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-gmblh"
Jul 06 20:43:08.940009 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.939986    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rl7xw\""
Jul 06 20:43:08.954979 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.952514    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-gmblh"]
Jul 06 20:43:08.990784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:08.990726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:09.030322 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.030272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zncj\" (UniqueName: \"kubernetes.io/projected/5caebcdc-62e3-46d8-a927-d0b968960429-kube-api-access-2zncj\") pod \"authorino-operator-6d85f6564-gmblh\" (UID: \"5caebcdc-62e3-46d8-a927-d0b968960429\") " pod="kuadrant-system/authorino-operator-6d85f6564-gmblh"
Jul 06 20:43:09.131587 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.131553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zncj\" (UniqueName: \"kubernetes.io/projected/5caebcdc-62e3-46d8-a927-d0b968960429-kube-api-access-2zncj\") pod \"authorino-operator-6d85f6564-gmblh\" (UID: \"5caebcdc-62e3-46d8-a927-d0b968960429\") " pod="kuadrant-system/authorino-operator-6d85f6564-gmblh"
Jul 06 20:43:09.160770 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.160733    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zncj\" (UniqueName: \"kubernetes.io/projected/5caebcdc-62e3-46d8-a927-d0b968960429-kube-api-access-2zncj\") pod \"authorino-operator-6d85f6564-gmblh\" (UID: \"5caebcdc-62e3-46d8-a927-d0b968960429\") " pod="kuadrant-system/authorino-operator-6d85f6564-gmblh"
Jul 06 20:43:09.180901 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.180871    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:09.246239 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.246160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-gmblh"
Jul 06 20:43:09.412766 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.412734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:09.433402 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.433371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-gmblh"]
Jul 06 20:43:09.434586 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:43:09.434559    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5caebcdc_62e3_46d8_a927_d0b968960429.slice/crio-318731da2e2884d99277445aa81477503dca60a42f60a94206d6f6f298c9c77d WatchSource:0}: Error finding container 318731da2e2884d99277445aa81477503dca60a42f60a94206d6f6f298c9c77d: Status 404 returned error can't find the container with id 318731da2e2884d99277445aa81477503dca60a42f60a94206d6f6f298c9c77d
Jul 06 20:43:09.497896 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.497809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-gmblh" event={"ID":"5caebcdc-62e3-46d8-a927-d0b968960429","Type":"ContainerStarted","Data":"318731da2e2884d99277445aa81477503dca60a42f60a94206d6f6f298c9c77d"}
Jul 06 20:43:09.794293 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:09.794202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:10.191947 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:10.191918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:11.439236 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:11.439198    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:13.085521 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.085485    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"]
Jul 06 20:43:13.090372 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.090351    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.093804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.093788    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w2qlv\""
Jul 06 20:43:13.108817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.108790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"]
Jul 06 20:43:13.165046 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.165014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff329fb-f3db-4c74-816f-a5882742d6f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw\" (UID: \"eff329fb-f3db-4c74-816f-a5882742d6f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.165207 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.165066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqrj\" (UniqueName: \"kubernetes.io/projected/eff329fb-f3db-4c74-816f-a5882742d6f0-kube-api-access-dtqrj\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw\" (UID: \"eff329fb-f3db-4c74-816f-a5882742d6f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.265540 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.265509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff329fb-f3db-4c74-816f-a5882742d6f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw\" (UID: \"eff329fb-f3db-4c74-816f-a5882742d6f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.265740 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.265557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqrj\" (UniqueName: \"kubernetes.io/projected/eff329fb-f3db-4c74-816f-a5882742d6f0-kube-api-access-dtqrj\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw\" (UID: \"eff329fb-f3db-4c74-816f-a5882742d6f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.265985 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.265935    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff329fb-f3db-4c74-816f-a5882742d6f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw\" (UID: \"eff329fb-f3db-4c74-816f-a5882742d6f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.283675 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.283652    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqrj\" (UniqueName: \"kubernetes.io/projected/eff329fb-f3db-4c74-816f-a5882742d6f0-kube-api-access-dtqrj\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw\" (UID: \"eff329fb-f3db-4c74-816f-a5882742d6f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.401131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.401039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:13.570608 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.570584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"]
Jul 06 20:43:13.570881 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:43:13.570860    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff329fb_f3db_4c74_816f_a5882742d6f0.slice/crio-9d422de1b9a3a34b47948b3ae1792ffd43e2fd9674cb05e8e441d1f5ba63a5b9 WatchSource:0}: Error finding container 9d422de1b9a3a34b47948b3ae1792ffd43e2fd9674cb05e8e441d1f5ba63a5b9: Status 404 returned error can't find the container with id 9d422de1b9a3a34b47948b3ae1792ffd43e2fd9674cb05e8e441d1f5ba63a5b9
Jul 06 20:43:13.634025 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.633996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:13.827836 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:13.827788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:14.527286 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:14.527233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw" event={"ID":"eff329fb-f3db-4c74-816f-a5882742d6f0","Type":"ContainerStarted","Data":"9d422de1b9a3a34b47948b3ae1792ffd43e2fd9674cb05e8e441d1f5ba63a5b9"}
Jul 06 20:43:15.635818 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.635780    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"]
Jul 06 20:43:15.643029 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.643000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"
Jul 06 20:43:15.647990 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.647948    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 06 20:43:15.648126 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.647948    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jhbn4\""
Jul 06 20:43:15.658608 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.658537    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"]
Jul 06 20:43:15.687953 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.687887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwnkb\" (UniqueName: \"kubernetes.io/projected/5ef4aac2-3fc7-46b5-8942-a8a7e46d8841-kube-api-access-pwnkb\") pod \"dns-operator-controller-manager-65b49595d7-wxznh\" (UID: \"5ef4aac2-3fc7-46b5-8942-a8a7e46d8841\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"
Jul 06 20:43:15.789140 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.789095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwnkb\" (UniqueName: \"kubernetes.io/projected/5ef4aac2-3fc7-46b5-8942-a8a7e46d8841-kube-api-access-pwnkb\") pod \"dns-operator-controller-manager-65b49595d7-wxznh\" (UID: \"5ef4aac2-3fc7-46b5-8942-a8a7e46d8841\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"
Jul 06 20:43:15.805686 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.805652    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwnkb\" (UniqueName: \"kubernetes.io/projected/5ef4aac2-3fc7-46b5-8942-a8a7e46d8841-kube-api-access-pwnkb\") pod \"dns-operator-controller-manager-65b49595d7-wxznh\" (UID: \"5ef4aac2-3fc7-46b5-8942-a8a7e46d8841\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"
Jul 06 20:43:15.955192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:15.955159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"
Jul 06 20:43:16.454895 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:16.454864    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"]
Jul 06 20:43:16.457757 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:43:16.457697    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef4aac2_3fc7_46b5_8942_a8a7e46d8841.slice/crio-8ffe8f4ef24a7b646422e3bb4e2b30febd07e50e7f505c9653c332d0b488d80a WatchSource:0}: Error finding container 8ffe8f4ef24a7b646422e3bb4e2b30febd07e50e7f505c9653c332d0b488d80a: Status 404 returned error can't find the container with id 8ffe8f4ef24a7b646422e3bb4e2b30febd07e50e7f505c9653c332d0b488d80a
Jul 06 20:43:16.541066 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:16.540946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh" event={"ID":"5ef4aac2-3fc7-46b5-8942-a8a7e46d8841","Type":"ContainerStarted","Data":"8ffe8f4ef24a7b646422e3bb4e2b30febd07e50e7f505c9653c332d0b488d80a"}
Jul 06 20:43:16.604124 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:16.604086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:16.716028 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:16.715921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:18.555588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:18.555436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh" event={"ID":"5ef4aac2-3fc7-46b5-8942-a8a7e46d8841","Type":"ContainerStarted","Data":"61c89e735864c5d8dd5b42476a4e63a35344b0da3299c91e5cca3789a01c0e83"}
Jul 06 20:43:18.555588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:18.555507    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"
Jul 06 20:43:18.557345 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:18.557318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-gmblh" event={"ID":"5caebcdc-62e3-46d8-a927-d0b968960429","Type":"ContainerStarted","Data":"ec2a851da02ab4a6dc81f27aa4d32fe290dd24f78f775155a0646d6efcff7dfd"}
Jul 06 20:43:18.557465 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:18.557447    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6d85f6564-gmblh"
Jul 06 20:43:18.597092 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:18.597026    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh" podStartSLOduration=1.847906564 podStartE2EDuration="3.597004034s" totalImagesPullingTime="1.74909747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:18.596376857 +0000 UTC m=+495.930717472" watchObservedRunningTime="2026-07-06 20:43:18.597004034 +0000 UTC m=+495.931344630"
Jul 06 20:43:18.630899 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:18.630843    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6d85f6564-gmblh" podStartSLOduration=2.530813204 podStartE2EDuration="10.630826153s" totalImagesPullingTime="8.100012949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:18.629260121 +0000 UTC m=+495.963600716" watchObservedRunningTime="2026-07-06 20:43:18.630826153 +0000 UTC m=+495.965166748"
Jul 06 20:43:19.501072 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:19.501038    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-s2frj"
Jul 06 20:43:20.557508 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:20.557475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:20.571391 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:20.571360    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw" event={"ID":"eff329fb-f3db-4c74-816f-a5882742d6f0","Type":"ContainerStarted","Data":"7499504f156c1590f39f57d305884c2b15c3d5da3e69d5887a8b7f1909c4c355"}
Jul 06 20:43:20.571534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:20.571449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:20.652146 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:20.652092    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw" podStartSLOduration=1.491218914 podStartE2EDuration="7.652075938s" totalImagesPullingTime="6.160857024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:20.646132863 +0000 UTC m=+497.980473469" watchObservedRunningTime="2026-07-06 20:43:20.652075938 +0000 UTC m=+497.986416533"
Jul 06 20:43:20.740561 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:20.740525    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:20.900107 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:20.900016    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:22.271650 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.271607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:22.272036 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.271898    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"]
Jul 06 20:43:22.275492 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.275476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.283478 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.283459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fq5bx\""
Jul 06 20:43:22.284118 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.284097    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 06 20:43:22.291263 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.291246    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 06 20:43:22.298268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.298239    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"]
Jul 06 20:43:22.349524 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.349492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2dd7826b-f568-4d72-8b11-c660ce6ea932-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.349524 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.349526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dd7826b-f568-4d72-8b11-c660ce6ea932-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.349722 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.349565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96jg\" (UniqueName: \"kubernetes.io/projected/2dd7826b-f568-4d72-8b11-c660ce6ea932-kube-api-access-h96jg\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.450273 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.450233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2dd7826b-f568-4d72-8b11-c660ce6ea932-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.450273 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.450284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dd7826b-f568-4d72-8b11-c660ce6ea932-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.450541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.450378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96jg\" (UniqueName: \"kubernetes.io/projected/2dd7826b-f568-4d72-8b11-c660ce6ea932-kube-api-access-h96jg\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.450541 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:43:22.450456    2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 06 20:43:22.450701 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:43:22.450686    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd7826b-f568-4d72-8b11-c660ce6ea932-plugin-serving-cert podName:2dd7826b-f568-4d72-8b11-c660ce6ea932 nodeName:}" failed. No retries permitted until 2026-07-06 20:43:22.950667004 +0000 UTC m=+500.285007577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2dd7826b-f568-4d72-8b11-c660ce6ea932-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-7jl5x" (UID: "2dd7826b-f568-4d72-8b11-c660ce6ea932") : secret "plugin-serving-cert" not found
Jul 06 20:43:22.450954 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.450934    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2dd7826b-f568-4d72-8b11-c660ce6ea932-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.462110 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.462084    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96jg\" (UniqueName: \"kubernetes.io/projected/2dd7826b-f568-4d72-8b11-c660ce6ea932-kube-api-access-h96jg\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.912234 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.912198    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:22.955539 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.955504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dd7826b-f568-4d72-8b11-c660ce6ea932-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:22.958023 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:22.957998    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dd7826b-f568-4d72-8b11-c660ce6ea932-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-7jl5x\" (UID: \"2dd7826b-f568-4d72-8b11-c660ce6ea932\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:23.184987 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:23.184940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"
Jul 06 20:43:23.337792 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:23.337766    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x"]
Jul 06 20:43:23.338525 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:43:23.338499    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dd7826b_f568_4d72_8b11_c660ce6ea932.slice/crio-fd39f0fa16cb48874a0d967cfc522f5fdd1c522b739d85c8bc444954120e588b WatchSource:0}: Error finding container fd39f0fa16cb48874a0d967cfc522f5fdd1c522b739d85c8bc444954120e588b: Status 404 returned error can't find the container with id fd39f0fa16cb48874a0d967cfc522f5fdd1c522b739d85c8bc444954120e588b
Jul 06 20:43:23.588028 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:23.587933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x" event={"ID":"2dd7826b-f568-4d72-8b11-c660ce6ea932","Type":"ContainerStarted","Data":"fd39f0fa16cb48874a0d967cfc522f5fdd1c522b739d85c8bc444954120e588b"}
Jul 06 20:43:25.721684 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:25.721645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:25.855635 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:25.855602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:25.988895 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:25.988824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:28.617113 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:28.617078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x" event={"ID":"2dd7826b-f568-4d72-8b11-c660ce6ea932","Type":"ContainerStarted","Data":"0175eeeb360440564a5028800f6842bd222d78341f370df00bd2eb50e0cd43c2"}
Jul 06 20:43:28.656431 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:28.656380    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-7jl5x" podStartSLOduration=2.230102431 podStartE2EDuration="6.65636628s" totalImagesPullingTime="4.426263849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:28.655924215 +0000 UTC m=+505.990264809" watchObservedRunningTime="2026-07-06 20:43:28.65636628 +0000 UTC m=+505.990706871"
Jul 06 20:43:29.567626 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:29.567595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6d85f6564-gmblh"
Jul 06 20:43:29.567782 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:29.567667    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-wxznh"
Jul 06 20:43:31.579041 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:31.579013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw"
Jul 06 20:43:32.449263 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:32.449227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:32.754855 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:32.754776    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:36.576485 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:36.576442    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:36.683097 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:36.683068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:40.396610 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:40.396575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:40.481886 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:40.481846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:40.574298 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:40.574260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:40.673808 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:40.673728    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:45.991160 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:45.991130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:46.059130 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:46.059098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:46.251190 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:46.251106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:46.345652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:46.345623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:46.434413 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:46.434379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:52.393013 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:52.392983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:43:52.495134 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:43:52.495105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:06.123049 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.122952    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"]
Jul 06 20:44:06.127682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.127656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:06.142878 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.142852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.146683 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.146648    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 20:44:06.149562 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.149482    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"]
Jul 06 20:44:06.232238 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.232203    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"]
Jul 06 20:44:06.233184 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.233160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0afb803d-2ade-4478-9a72-1436b4f542d7-config-file\") pod \"limitador-limitador-ff68ff5cb-6wblb\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.233302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.233276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzl4\" (UniqueName: \"kubernetes.io/projected/0afb803d-2ade-4478-9a72-1436b4f542d7-kube-api-access-xjzl4\") pod \"limitador-limitador-ff68ff5cb-6wblb\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.253484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.253456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:06.334025 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.333992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzl4\" (UniqueName: \"kubernetes.io/projected/0afb803d-2ade-4478-9a72-1436b4f542d7-kube-api-access-xjzl4\") pod \"limitador-limitador-ff68ff5cb-6wblb\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.334210 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.334049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0afb803d-2ade-4478-9a72-1436b4f542d7-config-file\") pod \"limitador-limitador-ff68ff5cb-6wblb\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.334646 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.334626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0afb803d-2ade-4478-9a72-1436b4f542d7-config-file\") pod \"limitador-limitador-ff68ff5cb-6wblb\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.354141 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.354111    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzl4\" (UniqueName: \"kubernetes.io/projected/0afb803d-2ade-4478-9a72-1436b4f542d7-kube-api-access-xjzl4\") pod \"limitador-limitador-ff68ff5cb-6wblb\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.360590 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.360557    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:06.463917 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.463875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:06.624522 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.624496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"]
Jul 06 20:44:06.626661 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:44:06.626637    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afb803d_2ade_4478_9a72_1436b4f542d7.slice/crio-96c9676efc3d821df902a929621ecc90bfdfca6e72948d0c82b20423e68b8d48 WatchSource:0}: Error finding container 96c9676efc3d821df902a929621ecc90bfdfca6e72948d0c82b20423e68b8d48: Status 404 returned error can't find the container with id 96c9676efc3d821df902a929621ecc90bfdfca6e72948d0c82b20423e68b8d48
Jul 06 20:44:06.815113 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:06.815032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb" event={"ID":"0afb803d-2ade-4478-9a72-1436b4f542d7","Type":"ContainerStarted","Data":"96c9676efc3d821df902a929621ecc90bfdfca6e72948d0c82b20423e68b8d48"}
Jul 06 20:44:07.121000 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.120862    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-sxwsz"]
Jul 06 20:44:07.125729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.125705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-sxwsz"
Jul 06 20:44:07.129293 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.129272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v6mxf\""
Jul 06 20:44:07.134616 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.134595    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-sxwsz"]
Jul 06 20:44:07.141074 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.141047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n24cq\" (UniqueName: \"kubernetes.io/projected/5596597c-c39e-4ef0-a153-3a2ff0ad396f-kube-api-access-n24cq\") pod \"authorino-b6c89bb85-sxwsz\" (UID: \"5596597c-c39e-4ef0-a153-3a2ff0ad396f\") " pod="kuadrant-system/authorino-b6c89bb85-sxwsz"
Jul 06 20:44:07.162295 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.162261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:07.241689 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.241656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n24cq\" (UniqueName: \"kubernetes.io/projected/5596597c-c39e-4ef0-a153-3a2ff0ad396f-kube-api-access-n24cq\") pod \"authorino-b6c89bb85-sxwsz\" (UID: \"5596597c-c39e-4ef0-a153-3a2ff0ad396f\") " pod="kuadrant-system/authorino-b6c89bb85-sxwsz"
Jul 06 20:44:07.252277 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.252248    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n24cq\" (UniqueName: \"kubernetes.io/projected/5596597c-c39e-4ef0-a153-3a2ff0ad396f-kube-api-access-n24cq\") pod \"authorino-b6c89bb85-sxwsz\" (UID: \"5596597c-c39e-4ef0-a153-3a2ff0ad396f\") " pod="kuadrant-system/authorino-b6c89bb85-sxwsz"
Jul 06 20:44:07.406312 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.406209    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-m28fv"]
Jul 06 20:44:07.408582 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.408554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:07.410682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.410503    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-m28fv"
Jul 06 20:44:07.426335 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.426310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-m28fv"]
Jul 06 20:44:07.436698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.436673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-sxwsz"
Jul 06 20:44:07.443029 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.442948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjbv\" (UniqueName: \"kubernetes.io/projected/300b8db5-9717-4e16-805d-14f7d180b46a-kube-api-access-gcjbv\") pod \"authorino-8df697c9f-m28fv\" (UID: \"300b8db5-9717-4e16-805d-14f7d180b46a\") " pod="kuadrant-system/authorino-8df697c9f-m28fv"
Jul 06 20:44:07.544094 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.543873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjbv\" (UniqueName: \"kubernetes.io/projected/300b8db5-9717-4e16-805d-14f7d180b46a-kube-api-access-gcjbv\") pod \"authorino-8df697c9f-m28fv\" (UID: \"300b8db5-9717-4e16-805d-14f7d180b46a\") " pod="kuadrant-system/authorino-8df697c9f-m28fv"
Jul 06 20:44:07.558935 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.558903    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjbv\" (UniqueName: \"kubernetes.io/projected/300b8db5-9717-4e16-805d-14f7d180b46a-kube-api-access-gcjbv\") pod \"authorino-8df697c9f-m28fv\" (UID: \"300b8db5-9717-4e16-805d-14f7d180b46a\") " pod="kuadrant-system/authorino-8df697c9f-m28fv"
Jul 06 20:44:07.599707 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.599679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-sxwsz"]
Jul 06 20:44:07.601748 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:44:07.601719    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5596597c_c39e_4ef0_a153_3a2ff0ad396f.slice/crio-db6b89b7326bc900fc9f8fde51efb85b75d4b33353a631aa93016337123ec0e5 WatchSource:0}: Error finding container db6b89b7326bc900fc9f8fde51efb85b75d4b33353a631aa93016337123ec0e5: Status 404 returned error can't find the container with id db6b89b7326bc900fc9f8fde51efb85b75d4b33353a631aa93016337123ec0e5
Jul 06 20:44:07.722076 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.722036    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-m28fv"
Jul 06 20:44:07.822870 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.822837    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-sxwsz" event={"ID":"5596597c-c39e-4ef0-a153-3a2ff0ad396f","Type":"ContainerStarted","Data":"db6b89b7326bc900fc9f8fde51efb85b75d4b33353a631aa93016337123ec0e5"}
Jul 06 20:44:07.874817 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:07.874783    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-m28fv"]
Jul 06 20:44:07.889753 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:44:07.889719    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod300b8db5_9717_4e16_805d_14f7d180b46a.slice/crio-9b749b8a25028913bad833c73a1b0a422213153f24968cd4651ef62af4b5b3b2 WatchSource:0}: Error finding container 9b749b8a25028913bad833c73a1b0a422213153f24968cd4651ef62af4b5b3b2: Status 404 returned error can't find the container with id 9b749b8a25028913bad833c73a1b0a422213153f24968cd4651ef62af4b5b3b2
Jul 06 20:44:08.587717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:08.587680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:08.834212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:08.834176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-m28fv" event={"ID":"300b8db5-9717-4e16-805d-14f7d180b46a","Type":"ContainerStarted","Data":"9b749b8a25028913bad833c73a1b0a422213153f24968cd4651ef62af4b5b3b2"}
Jul 06 20:44:12.859734 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.859701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb" event={"ID":"0afb803d-2ade-4478-9a72-1436b4f542d7","Type":"ContainerStarted","Data":"a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70"}
Jul 06 20:44:12.860185 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.859804    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:12.861332 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.861312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-m28fv" event={"ID":"300b8db5-9717-4e16-805d-14f7d180b46a","Type":"ContainerStarted","Data":"3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af"}
Jul 06 20:44:12.862685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.862666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-sxwsz" event={"ID":"5596597c-c39e-4ef0-a153-3a2ff0ad396f","Type":"ContainerStarted","Data":"f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658"}
Jul 06 20:44:12.892188 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.892131    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb" podStartSLOduration=1.351877331 podStartE2EDuration="6.892112747s" totalImagesPullingTime="5.540235416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:12.888686546 +0000 UTC m=+550.223027135" watchObservedRunningTime="2026-07-06 20:44:12.892112747 +0000 UTC m=+550.226453344"
Jul 06 20:44:12.910668 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.910604    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-m28fv" podStartSLOduration=1.647491053 podStartE2EDuration="5.910589644s" totalImagesPullingTime="4.263098591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:12.910344907 +0000 UTC m=+550.244685494" watchObservedRunningTime="2026-07-06 20:44:12.910589644 +0000 UTC m=+550.244930230"
Jul 06 20:44:12.934670 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.934618    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-sxwsz" podStartSLOduration=1.429089843 podStartE2EDuration="5.934602457s" totalImagesPullingTime="4.505512614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:12.933054931 +0000 UTC m=+550.267395530" watchObservedRunningTime="2026-07-06 20:44:12.934602457 +0000 UTC m=+550.268943052"
Jul 06 20:44:12.958689 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:12.958655    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-sxwsz"]
Jul 06 20:44:14.872927 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:14.872861    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-sxwsz" podUID="5596597c-c39e-4ef0-a153-3a2ff0ad396f" containerName="authorino" containerID="cri-o://f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658" gracePeriod=30
Jul 06 20:44:15.125193 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.125141    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-sxwsz"
Jul 06 20:44:15.220922 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.220889    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n24cq\" (UniqueName: \"kubernetes.io/projected/5596597c-c39e-4ef0-a153-3a2ff0ad396f-kube-api-access-n24cq\") pod \"5596597c-c39e-4ef0-a153-3a2ff0ad396f\" (UID: \"5596597c-c39e-4ef0-a153-3a2ff0ad396f\") "
Jul 06 20:44:15.223063 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.223033    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5596597c-c39e-4ef0-a153-3a2ff0ad396f-kube-api-access-n24cq" (OuterVolumeSpecName: "kube-api-access-n24cq") pod "5596597c-c39e-4ef0-a153-3a2ff0ad396f" (UID: "5596597c-c39e-4ef0-a153-3a2ff0ad396f"). InnerVolumeSpecName "kube-api-access-n24cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:44:15.321891 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.321854    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n24cq\" (UniqueName: \"kubernetes.io/projected/5596597c-c39e-4ef0-a153-3a2ff0ad396f-kube-api-access-n24cq\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:44:15.879501 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.879468    2582 generic.go:356] "Generic (PLEG): container finished" podID="5596597c-c39e-4ef0-a153-3a2ff0ad396f" containerID="f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658" exitCode=0
Jul 06 20:44:15.879900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.879513    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-sxwsz"
Jul 06 20:44:15.879900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.879547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-sxwsz" event={"ID":"5596597c-c39e-4ef0-a153-3a2ff0ad396f","Type":"ContainerDied","Data":"f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658"}
Jul 06 20:44:15.879900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.879588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-sxwsz" event={"ID":"5596597c-c39e-4ef0-a153-3a2ff0ad396f","Type":"ContainerDied","Data":"db6b89b7326bc900fc9f8fde51efb85b75d4b33353a631aa93016337123ec0e5"}
Jul 06 20:44:15.879900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.879604    2582 scope.go:117] "RemoveContainer" containerID="f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658"
Jul 06 20:44:15.888344 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.888327    2582 scope.go:117] "RemoveContainer" containerID="f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658"
Jul 06 20:44:15.888595 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:44:15.888577    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658\": container with ID starting with f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658 not found: ID does not exist" containerID="f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658"
Jul 06 20:44:15.888642 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.888604    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658"} err="failed to get container status \"f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658\": rpc error: code = NotFound desc = could not find container \"f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658\": container with ID starting with f53c8292662fd9b7c4b46111881920f86e75e3f90c920a48377c196053c30658 not found: ID does not exist"
Jul 06 20:44:15.908357 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.908335    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-sxwsz"]
Jul 06 20:44:15.911588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:15.911566    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-sxwsz"]
Jul 06 20:44:17.337920 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:17.337883    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5596597c-c39e-4ef0-a153-3a2ff0ad396f" path="/var/lib/kubelet/pods/5596597c-c39e-4ef0-a153-3a2ff0ad396f/volumes"
Jul 06 20:44:17.988454 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:17.988421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:18.099827 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:18.099792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:21.677471 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:21.677438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:21.890957 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:21.890923    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"]
Jul 06 20:44:21.891159 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:21.891136    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb" podUID="0afb803d-2ade-4478-9a72-1436b4f542d7" containerName="limitador" containerID="cri-o://a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70" gracePeriod=30
Jul 06 20:44:21.893521 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:21.893503    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:22.451988 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.451914    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:22.585649 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.585618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjzl4\" (UniqueName: \"kubernetes.io/projected/0afb803d-2ade-4478-9a72-1436b4f542d7-kube-api-access-xjzl4\") pod \"0afb803d-2ade-4478-9a72-1436b4f542d7\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") "
Jul 06 20:44:22.585826 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.585656    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0afb803d-2ade-4478-9a72-1436b4f542d7-config-file\") pod \"0afb803d-2ade-4478-9a72-1436b4f542d7\" (UID: \"0afb803d-2ade-4478-9a72-1436b4f542d7\") "
Jul 06 20:44:22.586077 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.586049    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afb803d-2ade-4478-9a72-1436b4f542d7-config-file" (OuterVolumeSpecName: "config-file") pod "0afb803d-2ade-4478-9a72-1436b4f542d7" (UID: "0afb803d-2ade-4478-9a72-1436b4f542d7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:44:22.587638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.587619    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afb803d-2ade-4478-9a72-1436b4f542d7-kube-api-access-xjzl4" (OuterVolumeSpecName: "kube-api-access-xjzl4") pod "0afb803d-2ade-4478-9a72-1436b4f542d7" (UID: "0afb803d-2ade-4478-9a72-1436b4f542d7"). InnerVolumeSpecName "kube-api-access-xjzl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:44:22.686265 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.686215    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjzl4\" (UniqueName: \"kubernetes.io/projected/0afb803d-2ade-4478-9a72-1436b4f542d7-kube-api-access-xjzl4\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:44:22.686265 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.686260    2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0afb803d-2ade-4478-9a72-1436b4f542d7-config-file\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:44:22.918387 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.918295    2582 generic.go:356] "Generic (PLEG): container finished" podID="0afb803d-2ade-4478-9a72-1436b4f542d7" containerID="a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70" exitCode=0
Jul 06 20:44:22.918387 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.918345    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"
Jul 06 20:44:22.918625 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.918379    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb" event={"ID":"0afb803d-2ade-4478-9a72-1436b4f542d7","Type":"ContainerDied","Data":"a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70"}
Jul 06 20:44:22.918625 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.918459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-6wblb" event={"ID":"0afb803d-2ade-4478-9a72-1436b4f542d7","Type":"ContainerDied","Data":"96c9676efc3d821df902a929621ecc90bfdfca6e72948d0c82b20423e68b8d48"}
Jul 06 20:44:22.918625 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.918483    2582 scope.go:117] "RemoveContainer" containerID="a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70"
Jul 06 20:44:22.927909 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.927892    2582 scope.go:117] "RemoveContainer" containerID="a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70"
Jul 06 20:44:22.928270 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:44:22.928250    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70\": container with ID starting with a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70 not found: ID does not exist" containerID="a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70"
Jul 06 20:44:22.928344 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.928283    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70"} err="failed to get container status \"a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70\": rpc error: code = NotFound desc = could not find container \"a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70\": container with ID starting with a64b9e466f5b4064151e5df9a01482172174569c32e1d868464fbe041d653f70 not found: ID does not exist"
Jul 06 20:44:22.941993 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.941952    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"]
Jul 06 20:44:22.947190 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:22.947166    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-6wblb"]
Jul 06 20:44:23.337107 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:23.337075    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0afb803d-2ade-4478-9a72-1436b4f542d7" path="/var/lib/kubelet/pods/0afb803d-2ade-4478-9a72-1436b4f542d7/volumes"
Jul 06 20:44:23.536030 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:23.536000    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:28.001623 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:28.001591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:28.482727 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:28.482686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:44:32.519094 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.519063    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-z4dhg"]
Jul 06 20:44:32.519486 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.519422    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5596597c-c39e-4ef0-a153-3a2ff0ad396f" containerName="authorino"
Jul 06 20:44:32.519486 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.519432    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5596597c-c39e-4ef0-a153-3a2ff0ad396f" containerName="authorino"
Jul 06 20:44:32.519486 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.519451    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0afb803d-2ade-4478-9a72-1436b4f542d7" containerName="limitador"
Jul 06 20:44:32.519486 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.519457    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0afb803d-2ade-4478-9a72-1436b4f542d7" containerName="limitador"
Jul 06 20:44:32.519642 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.519511    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5596597c-c39e-4ef0-a153-3a2ff0ad396f" containerName="authorino"
Jul 06 20:44:32.519642 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.519524    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0afb803d-2ade-4478-9a72-1436b4f542d7" containerName="limitador"
Jul 06 20:44:32.522575 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.522558    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.526503 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.526483    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 06 20:44:32.536553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.536533    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-z4dhg"]
Jul 06 20:44:32.679380 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.679341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/452f3ebb-1fc9-4ea2-80c1-bd0508dc7708-tls-cert\") pod \"authorino-7c77957575-z4dhg\" (UID: \"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708\") " pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.679569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.679399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjc4\" (UniqueName: \"kubernetes.io/projected/452f3ebb-1fc9-4ea2-80c1-bd0508dc7708-kube-api-access-whjc4\") pod \"authorino-7c77957575-z4dhg\" (UID: \"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708\") " pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.780722 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.780627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/452f3ebb-1fc9-4ea2-80c1-bd0508dc7708-tls-cert\") pod \"authorino-7c77957575-z4dhg\" (UID: \"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708\") " pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.780722 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.780693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjc4\" (UniqueName: \"kubernetes.io/projected/452f3ebb-1fc9-4ea2-80c1-bd0508dc7708-kube-api-access-whjc4\") pod \"authorino-7c77957575-z4dhg\" (UID: \"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708\") " pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.783119 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.783096    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/452f3ebb-1fc9-4ea2-80c1-bd0508dc7708-tls-cert\") pod \"authorino-7c77957575-z4dhg\" (UID: \"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708\") " pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.790439 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.790415    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjc4\" (UniqueName: \"kubernetes.io/projected/452f3ebb-1fc9-4ea2-80c1-bd0508dc7708-kube-api-access-whjc4\") pod \"authorino-7c77957575-z4dhg\" (UID: \"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708\") " pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.831325 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.831286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-z4dhg"
Jul 06 20:44:32.966863 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:32.966826    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-z4dhg"]
Jul 06 20:44:32.968571 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:44:32.968541    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod452f3ebb_1fc9_4ea2_80c1_bd0508dc7708.slice/crio-49e09b88b9990b6d21cdd6b52d03887a3c44cca5331525c7d9aad3b2c0aabbc5 WatchSource:0}: Error finding container 49e09b88b9990b6d21cdd6b52d03887a3c44cca5331525c7d9aad3b2c0aabbc5: Status 404 returned error can't find the container with id 49e09b88b9990b6d21cdd6b52d03887a3c44cca5331525c7d9aad3b2c0aabbc5
Jul 06 20:44:33.978261 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:33.978223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-z4dhg" event={"ID":"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708","Type":"ContainerStarted","Data":"814b0f1e4df1ac12409b260511142153fac113253dee1e57075d950022f18daf"}
Jul 06 20:44:33.978645 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:33.978281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-z4dhg" event={"ID":"452f3ebb-1fc9-4ea2-80c1-bd0508dc7708","Type":"ContainerStarted","Data":"49e09b88b9990b6d21cdd6b52d03887a3c44cca5331525c7d9aad3b2c0aabbc5"}
Jul 06 20:44:33.998900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:33.998849    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-z4dhg" podStartSLOduration=1.536934462 podStartE2EDuration="1.998834534s" totalImagesPullingTime="461.900072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:33.9969085 +0000 UTC m=+571.331249096" watchObservedRunningTime="2026-07-06 20:44:33.998834534 +0000 UTC m=+571.333175128"
Jul 06 20:44:34.028472 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.028441    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-m28fv"]
Jul 06 20:44:34.028634 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.028614    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-m28fv" podUID="300b8db5-9717-4e16-805d-14f7d180b46a" containerName="authorino" containerID="cri-o://3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af" gracePeriod=30
Jul 06 20:44:34.272691 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.272666    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-m28fv"
Jul 06 20:44:34.395447 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.395416    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcjbv\" (UniqueName: \"kubernetes.io/projected/300b8db5-9717-4e16-805d-14f7d180b46a-kube-api-access-gcjbv\") pod \"300b8db5-9717-4e16-805d-14f7d180b46a\" (UID: \"300b8db5-9717-4e16-805d-14f7d180b46a\") "
Jul 06 20:44:34.397428 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.397404    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300b8db5-9717-4e16-805d-14f7d180b46a-kube-api-access-gcjbv" (OuterVolumeSpecName: "kube-api-access-gcjbv") pod "300b8db5-9717-4e16-805d-14f7d180b46a" (UID: "300b8db5-9717-4e16-805d-14f7d180b46a"). InnerVolumeSpecName "kube-api-access-gcjbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:44:34.496209 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.496172    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcjbv\" (UniqueName: \"kubernetes.io/projected/300b8db5-9717-4e16-805d-14f7d180b46a-kube-api-access-gcjbv\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:44:34.984170 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.984130    2582 generic.go:356] "Generic (PLEG): container finished" podID="300b8db5-9717-4e16-805d-14f7d180b46a" containerID="3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af" exitCode=0
Jul 06 20:44:34.984619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.984177    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-m28fv"
Jul 06 20:44:34.984619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.984214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-m28fv" event={"ID":"300b8db5-9717-4e16-805d-14f7d180b46a","Type":"ContainerDied","Data":"3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af"}
Jul 06 20:44:34.984619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.984257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-m28fv" event={"ID":"300b8db5-9717-4e16-805d-14f7d180b46a","Type":"ContainerDied","Data":"9b749b8a25028913bad833c73a1b0a422213153f24968cd4651ef62af4b5b3b2"}
Jul 06 20:44:34.984619 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.984285    2582 scope.go:117] "RemoveContainer" containerID="3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af"
Jul 06 20:44:34.993000 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.992978    2582 scope.go:117] "RemoveContainer" containerID="3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af"
Jul 06 20:44:34.993231 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:44:34.993214    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af\": container with ID starting with 3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af not found: ID does not exist" containerID="3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af"
Jul 06 20:44:34.993294 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:34.993238    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af"} err="failed to get container status \"3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af\": rpc error: code = NotFound desc = could not find container \"3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af\": container with ID starting with 3d0fe4e1931366c36dd536032cd81d613bb49a3fc1b8d67983e75e690b5808af not found: ID does not exist"
Jul 06 20:44:35.010865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:35.010843    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-m28fv"]
Jul 06 20:44:35.012930 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:35.012913    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-m28fv"]
Jul 06 20:44:35.336456 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:35.336367    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="300b8db5-9717-4e16-805d-14f7d180b46a" path="/var/lib/kubelet/pods/300b8db5-9717-4e16-805d-14f7d180b46a/volumes"
Jul 06 20:44:38.428822 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.428786    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-2k89z"]
Jul 06 20:44:38.429301 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.429075    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z" podUID="712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" containerName="manager" containerID="cri-o://b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6" gracePeriod=10
Jul 06 20:44:38.455909 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.455884    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-7bghc"]
Jul 06 20:44:38.456271 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.456257    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="300b8db5-9717-4e16-805d-14f7d180b46a" containerName="authorino"
Jul 06 20:44:38.456313 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.456273    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="300b8db5-9717-4e16-805d-14f7d180b46a" containerName="authorino"
Jul 06 20:44:38.456355 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.456347    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="300b8db5-9717-4e16-805d-14f7d180b46a" containerName="authorino"
Jul 06 20:44:38.465698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.465671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.472462 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.472440    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-7bghc"]
Jul 06 20:44:38.632587 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.632554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b250379-0cfd-4a78-98db-a83fdc041069-cert\") pod \"kserve-controller-manager-68bc99fcb8-7bghc\" (UID: \"6b250379-0cfd-4a78-98db-a83fdc041069\") " pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.632733 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.632594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/6b250379-0cfd-4a78-98db-a83fdc041069-kube-api-access-jzpqz\") pod \"kserve-controller-manager-68bc99fcb8-7bghc\" (UID: \"6b250379-0cfd-4a78-98db-a83fdc041069\") " pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.685131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.685067    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:44:38.733443 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.733412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b250379-0cfd-4a78-98db-a83fdc041069-cert\") pod \"kserve-controller-manager-68bc99fcb8-7bghc\" (UID: \"6b250379-0cfd-4a78-98db-a83fdc041069\") " pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.733617 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.733452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/6b250379-0cfd-4a78-98db-a83fdc041069-kube-api-access-jzpqz\") pod \"kserve-controller-manager-68bc99fcb8-7bghc\" (UID: \"6b250379-0cfd-4a78-98db-a83fdc041069\") " pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.735741 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.735707    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b250379-0cfd-4a78-98db-a83fdc041069-cert\") pod \"kserve-controller-manager-68bc99fcb8-7bghc\" (UID: \"6b250379-0cfd-4a78-98db-a83fdc041069\") " pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.743531 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.743501    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpqz\" (UniqueName: \"kubernetes.io/projected/6b250379-0cfd-4a78-98db-a83fdc041069-kube-api-access-jzpqz\") pod \"kserve-controller-manager-68bc99fcb8-7bghc\" (UID: \"6b250379-0cfd-4a78-98db-a83fdc041069\") " pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.807222 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.807188    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:38.834308 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.834273    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert\") pod \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") "
Jul 06 20:44:38.834472 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.834398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzltf\" (UniqueName: \"kubernetes.io/projected/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-kube-api-access-fzltf\") pod \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\" (UID: \"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe\") "
Jul 06 20:44:38.836416 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.836388    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert" (OuterVolumeSpecName: "cert") pod "712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" (UID: "712eeea6-45b6-4fd4-bdcd-9a12b87d01fe"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:44:38.836521 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.836461    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-kube-api-access-fzltf" (OuterVolumeSpecName: "kube-api-access-fzltf") pod "712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" (UID: "712eeea6-45b6-4fd4-bdcd-9a12b87d01fe"). InnerVolumeSpecName "kube-api-access-fzltf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:44:38.935652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.935578    2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:44:38.935652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.935605    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzltf\" (UniqueName: \"kubernetes.io/projected/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe-kube-api-access-fzltf\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:44:38.972001 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:38.971956    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-7bghc"]
Jul 06 20:44:38.973272 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:44:38.973248    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b250379_0cfd_4a78_98db_a83fdc041069.slice/crio-0e9e595933501886fb328791c628ec61d3049fc52ac4988972f7af91dc7112b6 WatchSource:0}: Error finding container 0e9e595933501886fb328791c628ec61d3049fc52ac4988972f7af91dc7112b6: Status 404 returned error can't find the container with id 0e9e595933501886fb328791c628ec61d3049fc52ac4988972f7af91dc7112b6
Jul 06 20:44:39.008369 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.008338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc" event={"ID":"6b250379-0cfd-4a78-98db-a83fdc041069","Type":"ContainerStarted","Data":"0e9e595933501886fb328791c628ec61d3049fc52ac4988972f7af91dc7112b6"}
Jul 06 20:44:39.009705 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.009674    2582 generic.go:356] "Generic (PLEG): container finished" podID="712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" containerID="b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6" exitCode=0
Jul 06 20:44:39.009813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.009706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z" event={"ID":"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe","Type":"ContainerDied","Data":"b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6"}
Jul 06 20:44:39.009813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.009721    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z"
Jul 06 20:44:39.009813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.009751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-2k89z" event={"ID":"712eeea6-45b6-4fd4-bdcd-9a12b87d01fe","Type":"ContainerDied","Data":"9bdefa9daa60c9a8162a0bcf8aa583f095280eff040cfcc052edfdeb958b5df5"}
Jul 06 20:44:39.009813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.009777    2582 scope.go:117] "RemoveContainer" containerID="b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6"
Jul 06 20:44:39.018548 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.018524    2582 scope.go:117] "RemoveContainer" containerID="b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6"
Jul 06 20:44:39.018791 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:44:39.018773    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6\": container with ID starting with b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6 not found: ID does not exist" containerID="b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6"
Jul 06 20:44:39.018850 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.018799    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6"} err="failed to get container status \"b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6\": rpc error: code = NotFound desc = could not find container \"b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6\": container with ID starting with b3f979b47fb5272907666387f749da2d91c41bc8ca38bbc8ce209a1ccd9b8ab6 not found: ID does not exist"
Jul 06 20:44:39.033467 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.033442    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-2k89z"]
Jul 06 20:44:39.037119 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.037090    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-2k89z"]
Jul 06 20:44:39.337614 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:39.337587    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" path="/var/lib/kubelet/pods/712eeea6-45b6-4fd4-bdcd-9a12b87d01fe/volumes"
Jul 06 20:44:40.016040 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:40.016008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc" event={"ID":"6b250379-0cfd-4a78-98db-a83fdc041069","Type":"ContainerStarted","Data":"cddab80874cf2b279d8514184ea2183ffe92efa2a86926cc931236df2dc8e987"}
Jul 06 20:44:40.016477 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:40.016101    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:44:40.034564 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:44:40.034514    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc" podStartSLOduration=1.6808096510000001 podStartE2EDuration="2.034500611s" totalImagesPullingTime="353.69096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:40.03320404 +0000 UTC m=+577.367544649" watchObservedRunningTime="2026-07-06 20:44:40.034500611 +0000 UTC m=+577.368841209"
Jul 06 20:45:03.115733 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:03.115700    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:45:03.124034 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:03.124012    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:45:11.027379 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:11.027345    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc99fcb8-7bghc"
Jul 06 20:45:21.998643 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:21.998597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:22.049492 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.049460    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-s8blt"]
Jul 06 20:45:22.049912 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.049896    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" containerName="manager"
Jul 06 20:45:22.050069 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.049916    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" containerName="manager"
Jul 06 20:45:22.050069 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.050060    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="712eeea6-45b6-4fd4-bdcd-9a12b87d01fe" containerName="manager"
Jul 06 20:45:22.052246 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.052225    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.055748 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.055728    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 06 20:45:22.055844 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.055801    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-qd8r5\""
Jul 06 20:45:22.061766 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.061744    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zlrl9"]
Jul 06 20:45:22.064159 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.064142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.067611 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.067595    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 06 20:45:22.067696 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.067654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-lgjt6\""
Jul 06 20:45:22.074005 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.073983    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-s8blt"]
Jul 06 20:45:22.076665 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.076642    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zlrl9"]
Jul 06 20:45:22.102188 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.102143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:22.188466 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.188426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792e2b7d-169d-4b8f-9365-2126f95ab769-tls-certs\") pod \"model-serving-api-fd65d7d6-s8blt\" (UID: \"792e2b7d-169d-4b8f-9365-2126f95ab769\") " pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.188665 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.188524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mzx\" (UniqueName: \"kubernetes.io/projected/9915cda1-77b9-45e8-9882-feba99d6a970-kube-api-access-79mzx\") pod \"odh-model-controller-84b54b7f97-zlrl9\" (UID: \"9915cda1-77b9-45e8-9882-feba99d6a970\") " pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.188665 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.188570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbhc\" (UniqueName: \"kubernetes.io/projected/792e2b7d-169d-4b8f-9365-2126f95ab769-kube-api-access-wlbhc\") pod \"model-serving-api-fd65d7d6-s8blt\" (UID: \"792e2b7d-169d-4b8f-9365-2126f95ab769\") " pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.188665 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.188640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9915cda1-77b9-45e8-9882-feba99d6a970-cert\") pod \"odh-model-controller-84b54b7f97-zlrl9\" (UID: \"9915cda1-77b9-45e8-9882-feba99d6a970\") " pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.245702 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.245669    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:22.289534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.289454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9915cda1-77b9-45e8-9882-feba99d6a970-cert\") pod \"odh-model-controller-84b54b7f97-zlrl9\" (UID: \"9915cda1-77b9-45e8-9882-feba99d6a970\") " pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.289534 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.289504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792e2b7d-169d-4b8f-9365-2126f95ab769-tls-certs\") pod \"model-serving-api-fd65d7d6-s8blt\" (UID: \"792e2b7d-169d-4b8f-9365-2126f95ab769\") " pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.289714 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.289562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mzx\" (UniqueName: \"kubernetes.io/projected/9915cda1-77b9-45e8-9882-feba99d6a970-kube-api-access-79mzx\") pod \"odh-model-controller-84b54b7f97-zlrl9\" (UID: \"9915cda1-77b9-45e8-9882-feba99d6a970\") " pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.289714 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.289622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbhc\" (UniqueName: \"kubernetes.io/projected/792e2b7d-169d-4b8f-9365-2126f95ab769-kube-api-access-wlbhc\") pod \"model-serving-api-fd65d7d6-s8blt\" (UID: \"792e2b7d-169d-4b8f-9365-2126f95ab769\") " pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.289870 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:45:22.289853    2582 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 06 20:45:22.289953 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:45:22.289940    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/792e2b7d-169d-4b8f-9365-2126f95ab769-tls-certs podName:792e2b7d-169d-4b8f-9365-2126f95ab769 nodeName:}" failed. No retries permitted until 2026-07-06 20:45:22.789918751 +0000 UTC m=+620.124259342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/792e2b7d-169d-4b8f-9365-2126f95ab769-tls-certs") pod "model-serving-api-fd65d7d6-s8blt" (UID: "792e2b7d-169d-4b8f-9365-2126f95ab769") : secret "model-serving-api-tls" not found
Jul 06 20:45:22.292018 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.291995    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9915cda1-77b9-45e8-9882-feba99d6a970-cert\") pod \"odh-model-controller-84b54b7f97-zlrl9\" (UID: \"9915cda1-77b9-45e8-9882-feba99d6a970\") " pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.299239 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.299211    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mzx\" (UniqueName: \"kubernetes.io/projected/9915cda1-77b9-45e8-9882-feba99d6a970-kube-api-access-79mzx\") pod \"odh-model-controller-84b54b7f97-zlrl9\" (UID: \"9915cda1-77b9-45e8-9882-feba99d6a970\") " pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.299336 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.299264    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbhc\" (UniqueName: \"kubernetes.io/projected/792e2b7d-169d-4b8f-9365-2126f95ab769-kube-api-access-wlbhc\") pod \"model-serving-api-fd65d7d6-s8blt\" (UID: \"792e2b7d-169d-4b8f-9365-2126f95ab769\") " pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.373747 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.373691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:22.508109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.508084    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zlrl9"]
Jul 06 20:45:22.509835 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:45:22.509787    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9915cda1_77b9_45e8_9882_feba99d6a970.slice/crio-e5f2f4ba4abbdf6be8f631a43f45da37d97399d4cb7214a983f4245408d171dd WatchSource:0}: Error finding container e5f2f4ba4abbdf6be8f631a43f45da37d97399d4cb7214a983f4245408d171dd: Status 404 returned error can't find the container with id e5f2f4ba4abbdf6be8f631a43f45da37d97399d4cb7214a983f4245408d171dd
Jul 06 20:45:22.605544 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.605466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:22.776386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.776356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:22.793562 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.793538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792e2b7d-169d-4b8f-9365-2126f95ab769-tls-certs\") pod \"model-serving-api-fd65d7d6-s8blt\" (UID: \"792e2b7d-169d-4b8f-9365-2126f95ab769\") " pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.795909 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.795891    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792e2b7d-169d-4b8f-9365-2126f95ab769-tls-certs\") pod \"model-serving-api-fd65d7d6-s8blt\" (UID: \"792e2b7d-169d-4b8f-9365-2126f95ab769\") " pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:22.843799 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.843770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:22.962287 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:22.962248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:23.098275 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:23.098239    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-s8blt"]
Jul 06 20:45:23.100140 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:45:23.100113    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792e2b7d_169d_4b8f_9365_2126f95ab769.slice/crio-1ee4843712f9edb255074c7df7e3e94c743359de9f9e8fe0c69aaafb5fd3c4ee WatchSource:0}: Error finding container 1ee4843712f9edb255074c7df7e3e94c743359de9f9e8fe0c69aaafb5fd3c4ee: Status 404 returned error can't find the container with id 1ee4843712f9edb255074c7df7e3e94c743359de9f9e8fe0c69aaafb5fd3c4ee
Jul 06 20:45:23.253761 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:23.253667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-s8blt" event={"ID":"792e2b7d-169d-4b8f-9365-2126f95ab769","Type":"ContainerStarted","Data":"1ee4843712f9edb255074c7df7e3e94c743359de9f9e8fe0c69aaafb5fd3c4ee"}
Jul 06 20:45:23.255102 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:23.255067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zlrl9" event={"ID":"9915cda1-77b9-45e8-9882-feba99d6a970","Type":"ContainerStarted","Data":"e5f2f4ba4abbdf6be8f631a43f45da37d97399d4cb7214a983f4245408d171dd"}
Jul 06 20:45:26.276178 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:26.276143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-s8blt" event={"ID":"792e2b7d-169d-4b8f-9365-2126f95ab769","Type":"ContainerStarted","Data":"380cef472202b412b2c0b23bd7c0ea94e43ccb63fedd282eaf2ce45050ee846c"}
Jul 06 20:45:26.276625 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:26.276320    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:26.277597 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:26.277572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zlrl9" event={"ID":"9915cda1-77b9-45e8-9882-feba99d6a970","Type":"ContainerStarted","Data":"92607a741dfe9deb7ed2fd1cb20b5bbdbc5d93cc60db71c50eee7ffc5ee28e3b"}
Jul 06 20:45:26.277717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:26.277703    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:26.297574 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:26.297529    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-s8blt" podStartSLOduration=1.8014450659999999 podStartE2EDuration="4.297517451s" totalImagesPullingTime="2.496072385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:26.295319672 +0000 UTC m=+623.629660271" watchObservedRunningTime="2026-07-06 20:45:26.297517451 +0000 UTC m=+623.631858045"
Jul 06 20:45:26.312150 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:26.312109    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-zlrl9" podStartSLOduration=1.212933387 podStartE2EDuration="4.312095724s" totalImagesPullingTime="3.099162337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:26.311380042 +0000 UTC m=+623.645720637" watchObservedRunningTime="2026-07-06 20:45:26.312095724 +0000 UTC m=+623.646436318"
Jul 06 20:45:29.691909 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:29.691874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:29.792413 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:29.792382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:45:37.286281 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:37.286231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-zlrl9"
Jul 06 20:45:37.287980 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:37.287946    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-s8blt"
Jul 06 20:45:38.234686 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.234648    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vmx7n"]
Jul 06 20:45:38.238180 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.238164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.241558 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.241529    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fqzmq\""
Jul 06 20:45:38.241701 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.241567    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 06 20:45:38.249127 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.249101    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vmx7n"]
Jul 06 20:45:38.337027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.336993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-model-cache\") pod \"s3-init-vmx7n\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") " pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.337027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.337032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zv2\" (UniqueName: \"kubernetes.io/projected/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-kube-api-access-m5zv2\") pod \"s3-init-vmx7n\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") " pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.438115 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.438079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-model-cache\") pod \"s3-init-vmx7n\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") " pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.438325 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.438130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zv2\" (UniqueName: \"kubernetes.io/projected/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-kube-api-access-m5zv2\") pod \"s3-init-vmx7n\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") " pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.438552 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.438527    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-model-cache\") pod \"s3-init-vmx7n\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") " pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.447503 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.447474    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zv2\" (UniqueName: \"kubernetes.io/projected/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-kube-api-access-m5zv2\") pod \"s3-init-vmx7n\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") " pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.548587 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.548490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmx7n"
Jul 06 20:45:38.685797 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:38.685770    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vmx7n"]
Jul 06 20:45:38.687919 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:45:38.687884    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d05dfe9_43fd_4f2c_b62e_125fd10adce5.slice/crio-9e51b105b0740c298696826e8200ce5f02c5a39e1e44f0c8e271fa5b79c69f14 WatchSource:0}: Error finding container 9e51b105b0740c298696826e8200ce5f02c5a39e1e44f0c8e271fa5b79c69f14: Status 404 returned error can't find the container with id 9e51b105b0740c298696826e8200ce5f02c5a39e1e44f0c8e271fa5b79c69f14
Jul 06 20:45:39.348251 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:39.348212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmx7n" event={"ID":"6d05dfe9-43fd-4f2c-b62e-125fd10adce5","Type":"ContainerStarted","Data":"9e51b105b0740c298696826e8200ce5f02c5a39e1e44f0c8e271fa5b79c69f14"}
Jul 06 20:45:43.371673 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:43.371637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmx7n" event={"ID":"6d05dfe9-43fd-4f2c-b62e-125fd10adce5","Type":"ContainerStarted","Data":"c3c178af22591a95960e80cd20ab9b9275fff4d86edbc6c4e7035b10c56ca62b"}
Jul 06 20:45:45.383315 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:45.383233    2582 generic.go:356] "Generic (PLEG): container finished" podID="6d05dfe9-43fd-4f2c-b62e-125fd10adce5" containerID="c3c178af22591a95960e80cd20ab9b9275fff4d86edbc6c4e7035b10c56ca62b" exitCode=0
Jul 06 20:45:45.383667 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:45.383306    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmx7n" event={"ID":"6d05dfe9-43fd-4f2c-b62e-125fd10adce5","Type":"ContainerDied","Data":"c3c178af22591a95960e80cd20ab9b9275fff4d86edbc6c4e7035b10c56ca62b"}
Jul 06 20:45:51.421646 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:51.421606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmx7n" event={"ID":"6d05dfe9-43fd-4f2c-b62e-125fd10adce5","Type":"ContainerStarted","Data":"17465b360a90a4d937f8df7caeff0f88ff090093b4119eefcdbb4ae6b0ab25d6"}
Jul 06 20:45:51.443144 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:45:51.443087    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vmx7n" podStartSLOduration=2.237955234 podStartE2EDuration="13.443068152s" totalImagesPullingTime="9.091328778s" totalInitContainerRuntime="2.11378414s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:51.440764971 +0000 UTC m=+648.775105573" watchObservedRunningTime="2026-07-06 20:45:51.443068152 +0000 UTC m=+648.777408747"
Jul 06 20:46:22.378784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:22.378753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:44.814982 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:44.814932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:44.915781 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:44.915748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:49.884158 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:49.884109    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:55.762714 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:55.762683    2582 generic.go:356] "Generic (PLEG): container finished" podID="6d05dfe9-43fd-4f2c-b62e-125fd10adce5" containerID="17465b360a90a4d937f8df7caeff0f88ff090093b4119eefcdbb4ae6b0ab25d6" exitCode=0
Jul 06 20:46:55.763136 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:55.762749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmx7n" event={"ID":"6d05dfe9-43fd-4f2c-b62e-125fd10adce5","Type":"ContainerDied","Data":"17465b360a90a4d937f8df7caeff0f88ff090093b4119eefcdbb4ae6b0ab25d6"}
Jul 06 20:46:56.906653 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:56.906628    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmx7n"
Jul 06 20:46:57.045550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.045459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5zv2\" (UniqueName: \"kubernetes.io/projected/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-kube-api-access-m5zv2\") pod \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") "
Jul 06 20:46:57.045550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.045522    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-model-cache\") pod \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\" (UID: \"6d05dfe9-43fd-4f2c-b62e-125fd10adce5\") "
Jul 06 20:46:57.047582 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.047553    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-kube-api-access-m5zv2" (OuterVolumeSpecName: "kube-api-access-m5zv2") pod "6d05dfe9-43fd-4f2c-b62e-125fd10adce5" (UID: "6d05dfe9-43fd-4f2c-b62e-125fd10adce5"). InnerVolumeSpecName "kube-api-access-m5zv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:46:57.068669 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.068639    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d05dfe9-43fd-4f2c-b62e-125fd10adce5" (UID: "6d05dfe9-43fd-4f2c-b62e-125fd10adce5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:46:57.146405 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.146369    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5zv2\" (UniqueName: \"kubernetes.io/projected/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-kube-api-access-m5zv2\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:46:57.146405 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.146403    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d05dfe9-43fd-4f2c-b62e-125fd10adce5-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:46:57.774348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.774317    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmx7n"
Jul 06 20:46:57.774525 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.774347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmx7n" event={"ID":"6d05dfe9-43fd-4f2c-b62e-125fd10adce5","Type":"ContainerDied","Data":"9e51b105b0740c298696826e8200ce5f02c5a39e1e44f0c8e271fa5b79c69f14"}
Jul 06 20:46:57.774525 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:57.774384    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e51b105b0740c298696826e8200ce5f02c5a39e1e44f0c8e271fa5b79c69f14"
Jul 06 20:46:58.221497 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:58.221461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:46:58.320208 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:46:58.320173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:11.454977 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.454923    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"]
Jul 06 20:47:11.455445 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.455326    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d05dfe9-43fd-4f2c-b62e-125fd10adce5" containerName="download-hf-model"
Jul 06 20:47:11.455445 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.455338    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d05dfe9-43fd-4f2c-b62e-125fd10adce5" containerName="download-hf-model"
Jul 06 20:47:11.455445 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.455359    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d05dfe9-43fd-4f2c-b62e-125fd10adce5" containerName="s3-init"
Jul 06 20:47:11.455445 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.455364    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d05dfe9-43fd-4f2c-b62e-125fd10adce5" containerName="s3-init"
Jul 06 20:47:11.455445 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.455420    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d05dfe9-43fd-4f2c-b62e-125fd10adce5" containerName="s3-init"
Jul 06 20:47:11.457566 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.457548    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.461239 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.461216    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-67vjw\""
Jul 06 20:47:11.461375 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.461320    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 20:47:11.461450 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.461403    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 20:47:11.461537 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.461518    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 06 20:47:11.481114 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.481087    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"]
Jul 06 20:47:11.482701 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.482676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.482836 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.482722    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmwh\" (UniqueName: \"kubernetes.io/projected/ee146378-761d-47dd-96b5-24f9cfaff899-kube-api-access-8vmwh\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.482836 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.482753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee146378-761d-47dd-96b5-24f9cfaff899-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.482836 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.482817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.483001 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.482849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.483001 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.482931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee146378-761d-47dd-96b5-24f9cfaff899-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.483001 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.482996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.483156 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.483016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.483156 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.483030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee146378-761d-47dd-96b5-24f9cfaff899-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.501139 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.501108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:11.583955 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.583923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.583955 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.583958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee146378-761d-47dd-96b5-24f9cfaff899-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmwh\" (UniqueName: \"kubernetes.io/projected/ee146378-761d-47dd-96b5-24f9cfaff899-kube-api-access-8vmwh\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee146378-761d-47dd-96b5-24f9cfaff899-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee146378-761d-47dd-96b5-24f9cfaff899-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584389    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584627 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584574    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584675 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584651    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584719 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584669    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.584866 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.584843    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee146378-761d-47dd-96b5-24f9cfaff899-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.586481 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.586456    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee146378-761d-47dd-96b5-24f9cfaff899-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.586620 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.586606    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee146378-761d-47dd-96b5-24f9cfaff899-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.596702 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.596677    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee146378-761d-47dd-96b5-24f9cfaff899-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.597001 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.596980    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmwh\" (UniqueName: \"kubernetes.io/projected/ee146378-761d-47dd-96b5-24f9cfaff899-kube-api-access-8vmwh\") pod \"router-gateway-1-openshift-default-799f46c59b-pl8dd\" (UID: \"ee146378-761d-47dd-96b5-24f9cfaff899\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.603282 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.603257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:11.768051 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.767930    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:11.927989 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.927846    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"]
Jul 06 20:47:11.931072 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:47:11.931040    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee146378_761d_47dd_96b5_24f9cfaff899.slice/crio-070a9844c94ae1afc80a230298009b343e49623f3b81253177f7fa55d55aaa4f WatchSource:0}: Error finding container 070a9844c94ae1afc80a230298009b343e49623f3b81253177f7fa55d55aaa4f: Status 404 returned error can't find the container with id 070a9844c94ae1afc80a230298009b343e49623f3b81253177f7fa55d55aaa4f
Jul 06 20:47:11.933162 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.933124    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:47:11.933244 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.933188    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:47:11.933244 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:11.933216    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:47:12.865397 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:12.865356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerStarted","Data":"1632a8f30152453730f8a45642f50144b47a9906b7296f9ca74d6d6ef3178d74"}
Jul 06 20:47:12.865791 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:12.865422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerStarted","Data":"070a9844c94ae1afc80a230298009b343e49623f3b81253177f7fa55d55aaa4f"}
Jul 06 20:47:12.890989 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:12.890910    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podStartSLOduration=1.890893809 podStartE2EDuration="1.890893809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:12.890205247 +0000 UTC m=+730.224545843" watchObservedRunningTime="2026-07-06 20:47:12.890893809 +0000 UTC m=+730.225234403"
Jul 06 20:47:13.768789 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:13.768747    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:13.773677 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:13.773652    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:13.870627 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:13.870599    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:13.871660 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:13.871638    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:47:15.149889 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:15.149852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:22.387177 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.387141    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"]
Jul 06 20:47:22.389848 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.389832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.393645 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.393624    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:47:22.394798 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.394774    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 06 20:47:22.394798 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.394783    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 20:47:22.405243 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.405221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"]
Jul 06 20:47:22.432134 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.432103    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:22.473761 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.473732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tls-certs\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.473915 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.473767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-model-cache\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.473915 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.473802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.473915 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.473829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.473915 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.473856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2czx\" (UniqueName: \"kubernetes.io/projected/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kube-api-access-k2czx\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.473915 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.473874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-home\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.474149 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.473945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-dshm\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574463 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574629 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2czx\" (UniqueName: \"kubernetes.io/projected/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kube-api-access-k2czx\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574629 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-home\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574629 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-dshm\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574629 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tls-certs\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574842 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-model-cache\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574842 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574954 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574888    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.574954 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.574916    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-home\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.575080 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.575068    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-model-cache\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.575140 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.575119    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.576981 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.576941    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-dshm\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.577098 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.577052    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tls-certs\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.586525 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.586499    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2czx\" (UniqueName: \"kubernetes.io/projected/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kube-api-access-k2czx\") pod \"precise-prefix-cache-test-kserve-6d789b4bf5-5whz4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.701755 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.701719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:22.845420 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.845396    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"]
Jul 06 20:47:22.847485 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:47:22.847454    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf81f5b_d1ea_42e2_ad83_2534d60529d4.slice/crio-5c243621f22ba28a5320528ce0872822afc60055568c1c4a1939c6e87bcef0b0 WatchSource:0}: Error finding container 5c243621f22ba28a5320528ce0872822afc60055568c1c4a1939c6e87bcef0b0: Status 404 returned error can't find the container with id 5c243621f22ba28a5320528ce0872822afc60055568c1c4a1939c6e87bcef0b0
Jul 06 20:47:22.924061 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.923953    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"]
Jul 06 20:47:22.927334 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.927303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" event={"ID":"7bf81f5b-d1ea-42e2-ad83-2534d60529d4","Type":"ContainerStarted","Data":"19f882878c1f33e903f8ca09849e17844f61cdecf7102bb5b18f1ca70f0b3c90"}
Jul 06 20:47:22.927530 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.927346    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" event={"ID":"7bf81f5b-d1ea-42e2-ad83-2534d60529d4","Type":"ContainerStarted","Data":"5c243621f22ba28a5320528ce0872822afc60055568c1c4a1939c6e87bcef0b0"}
Jul 06 20:47:22.927530 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.927482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:22.931753 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.931234    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-wfbcx\""
Jul 06 20:47:22.950131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.950096    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"]
Jul 06 20:47:22.966195 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.966119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:22.978715 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.978682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:22.978852 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.978752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:22.978852 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.978799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:22.978951 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.978894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:22.979025 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.978976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:22.979071 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:22.979053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwkqw\" (UniqueName: \"kubernetes.io/projected/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kube-api-access-wwkqw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080424 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080600 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkqw\" (UniqueName: \"kubernetes.io/projected/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kube-api-access-wwkqw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080600 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080600 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080600 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080803 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080863 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080823    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.080926 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080883    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.081002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.080932    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.081057 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.081023    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.083002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.082975    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.090697 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.090676    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkqw\" (UniqueName: \"kubernetes.io/projected/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kube-api-access-wwkqw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.245779 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.245693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:47:23.421399 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.421182    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"]
Jul 06 20:47:23.424092 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:47:23.424042    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd908d052_8197_473b_bbd8_e5e9b8b3d7a7.slice/crio-4f3f9df12cbd1804546fa84c08bee4bd0a318c871bef77dc05619b6328ec279f WatchSource:0}: Error finding container 4f3f9df12cbd1804546fa84c08bee4bd0a318c871bef77dc05619b6328ec279f: Status 404 returned error can't find the container with id 4f3f9df12cbd1804546fa84c08bee4bd0a318c871bef77dc05619b6328ec279f
Jul 06 20:47:23.938538 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.938485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerStarted","Data":"04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735"}
Jul 06 20:47:23.938717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:23.938559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerStarted","Data":"4f3f9df12cbd1804546fa84c08bee4bd0a318c871bef77dc05619b6328ec279f"}
Jul 06 20:47:24.945116 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:24.945084    2582 generic.go:356] "Generic (PLEG): container finished" podID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerID="04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735" exitCode=0
Jul 06 20:47:24.945565 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:24.945164    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerDied","Data":"04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735"}
Jul 06 20:47:24.946226 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:24.946211    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:47:26.962569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:26.962527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerStarted","Data":"aed886fdda0e04f6d783e8450b2c69e71ee88456d4adbcefe6d71a5e4e03d0d8"}
Jul 06 20:47:27.971307 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:27.971265    2582 generic.go:356] "Generic (PLEG): container finished" podID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerID="19f882878c1f33e903f8ca09849e17844f61cdecf7102bb5b18f1ca70f0b3c90" exitCode=0
Jul 06 20:47:27.971807 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:27.971362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" event={"ID":"7bf81f5b-d1ea-42e2-ad83-2534d60529d4","Type":"ContainerDied","Data":"19f882878c1f33e903f8ca09849e17844f61cdecf7102bb5b18f1ca70f0b3c90"}
Jul 06 20:47:30.999577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:30.999416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" event={"ID":"7bf81f5b-d1ea-42e2-ad83-2534d60529d4","Type":"ContainerStarted","Data":"ceeeddbe1a849cfe2ffabbb52196be8e7ac89d62f18baf839ec77c978ebcaaaa"}
Jul 06 20:47:31.027064 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:31.026941    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podStartSLOduration=2.000465942 podStartE2EDuration="9.026921133s" totalImagesPullingTime="2.786851637s" totalInitContainerRuntime="4.239603554s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:31.022931028 +0000 UTC m=+748.357271672" watchObservedRunningTime="2026-07-06 20:47:31.026921133 +0000 UTC m=+748.361261729"
Jul 06 20:47:31.702062 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:31.702029    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:32.702554 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:32.702516    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:32.718556 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:32.718528    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:33.030104 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:33.030025    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:47:34.024700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:34.024664    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="b7e7f760ad637870ff4c58e43d1480f88326c91579dfa567ba3d521fb9f7ddf2" exitCode=0
Jul 06 20:47:34.025172 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:34.024746    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"b7e7f760ad637870ff4c58e43d1480f88326c91579dfa567ba3d521fb9f7ddf2"}
Jul 06 20:47:34.025422 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:34.025404    2582 scope.go:117] "RemoveContainer" containerID="b7e7f760ad637870ff4c58e43d1480f88326c91579dfa567ba3d521fb9f7ddf2"
Jul 06 20:47:34.026682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:34.026651    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:47:34.026792 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:34.026768    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:47:34.026861 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:34.026848    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:47:35.035767 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:35.035714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"2f068bd56567e2c311987dec49318f1aae94721ebc31038b46bcb24f1bcf3ea7"}
Jul 06 20:47:35.556519 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:35.556474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:47:35.557670 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:35.557636    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:47:35.557809 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:35.557708    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:47:36.556890 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:36.556852    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:47:36.557332 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:36.556931    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:47:37.051139 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:37.051108    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9_d908d052-8197-473b-bbd8-e5e9b8b3d7a7/main/0.log"
Jul 06 20:47:37.051581 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:37.051551    2582 generic.go:356] "Generic (PLEG): container finished" podID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerID="aed886fdda0e04f6d783e8450b2c69e71ee88456d4adbcefe6d71a5e4e03d0d8" exitCode=1
Jul 06 20:47:37.051700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:37.051647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerDied","Data":"aed886fdda0e04f6d783e8450b2c69e71ee88456d4adbcefe6d71a5e4e03d0d8"}
Jul 06 20:47:37.374364 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:37.374330    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:47:37.556896 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:37.556857    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:47:37.557295 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:37.556937    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:47:38.556106 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:38.556070    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:47:38.556310 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:38.556128    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:47:39.556289 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:39.556247    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:47:39.556758 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:39.556317    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:47:40.560977 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:40.560926    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:47:40.562278 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:40.562246    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:47:41.093450 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:41.093419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:41.158550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:41.158513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:44.896366 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:44.896292    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:47:58.872109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:58.872048    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 06 20:47:58.984611 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:47:58.984572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:00.468348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:00.468309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:02.237508 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:02.237477    2582 generic.go:356] "Generic (PLEG): container finished" podID="ee146378-761d-47dd-96b5-24f9cfaff899" containerID="1632a8f30152453730f8a45642f50144b47a9906b7296f9ca74d6d6ef3178d74" exitCode=0
Jul 06 20:48:02.237897 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:02.237549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerDied","Data":"1632a8f30152453730f8a45642f50144b47a9906b7296f9ca74d6d6ef3178d74"}
Jul 06 20:48:02.237998 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:02.237983    2582 scope.go:117] "RemoveContainer" containerID="1632a8f30152453730f8a45642f50144b47a9906b7296f9ca74d6d6ef3178d74"
Jul 06 20:48:02.275110 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:02.275065    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:48:02.275268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:02.275165    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:48:02.275268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:02.275212    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:48:03.245801 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:03.245752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerStarted","Data":"60cc2f8c9c8fc1f646c263943928c0eef23e8a38494e6ae63c6121eb5c3d58ed"}
Jul 06 20:48:03.247784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:03.247765    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9_d908d052-8197-473b-bbd8-e5e9b8b3d7a7/main/0.log"
Jul 06 20:48:03.248201 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:03.248176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerStarted","Data":"e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef"}
Jul 06 20:48:03.248421 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:03.248395    2582 scope.go:117] "RemoveContainer" containerID="aed886fdda0e04f6d783e8450b2c69e71ee88456d4adbcefe6d71a5e4e03d0d8"
Jul 06 20:48:03.769127 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:03.769090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:03.773292 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:03.773268    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:04.258225 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:04.258194    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9_d908d052-8197-473b-bbd8-e5e9b8b3d7a7/main/0.log"
Jul 06 20:48:04.258723 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:04.258606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerStarted","Data":"67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1"}
Jul 06 20:48:04.258794 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:04.258740    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:04.259063 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:04.259045    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:04.260054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:04.260036    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:04.285622 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:04.285570    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" podStartSLOduration=4.106184815 podStartE2EDuration="42.28555641s" totalImagesPullingTime="37.331003343s" totalInitContainerRuntime="848.368252ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:04.281743835 +0000 UTC m=+781.616084453" watchObservedRunningTime="2026-07-06 20:48:04.28555641 +0000 UTC m=+781.619897005"
Jul 06 20:48:13.246263 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:13.246211    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:13.246830 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:13.246342    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:13.247686 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:48:13.247658    2582 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:9003", ServerName: "10.134.0.48:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.48:9003: connect: connection refused"
Jul 06 20:48:13.249043 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:13.249023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:13.319282 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:13.319255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:14.246730 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:14.246686    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:9003\" within 1s: context deadline exceeded"
Jul 06 20:48:16.954221 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:16.954191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:20.679415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:20.679380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:23.247254 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:48:23.247225    2582 logging.go:55] [core] [Channel #19 SubChannel #20]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:9003", ServerName: "10.134.0.48:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.48:9003: connect: connection refused"
Jul 06 20:48:24.247163 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:24.247115    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:9003\" within 1s: context deadline exceeded"
Jul 06 20:48:27.405947 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:27.405913    2582 generic.go:356] "Generic (PLEG): container finished" podID="ee146378-761d-47dd-96b5-24f9cfaff899" containerID="60cc2f8c9c8fc1f646c263943928c0eef23e8a38494e6ae63c6121eb5c3d58ed" exitCode=0
Jul 06 20:48:27.406441 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:27.405991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerDied","Data":"60cc2f8c9c8fc1f646c263943928c0eef23e8a38494e6ae63c6121eb5c3d58ed"}
Jul 06 20:48:27.406441 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:27.406050    2582 scope.go:117] "RemoveContainer" containerID="1632a8f30152453730f8a45642f50144b47a9906b7296f9ca74d6d6ef3178d74"
Jul 06 20:48:27.406441 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:27.406433    2582 scope.go:117] "RemoveContainer" containerID="60cc2f8c9c8fc1f646c263943928c0eef23e8a38494e6ae63c6121eb5c3d58ed"
Jul 06 20:48:27.406670 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:48:27.406642    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899"
Jul 06 20:48:32.664391 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.664358    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"]
Jul 06 20:48:32.703107 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.703076    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"]
Jul 06 20:48:32.703305 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.703200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.707058 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.707030    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 06 20:48:32.742088 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.742062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:32.819516 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.819480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-model-cache\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.819711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.819528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-kserve-provision-location\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.819711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.819592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-tmp-dir\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.819711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.819614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64sn6\" (UniqueName: \"kubernetes.io/projected/f7a5c4a0-be20-4561-be9d-956c030894e7-kube-api-access-64sn6\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.819711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.819683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-dshm\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.819878 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.819729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a5c4a0-be20-4561-be9d-956c030894e7-tls-certs\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.819878 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.819828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-home\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-home\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-model-cache\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-kserve-provision-location\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-tmp-dir\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64sn6\" (UniqueName: \"kubernetes.io/projected/f7a5c4a0-be20-4561-be9d-956c030894e7-kube-api-access-64sn6\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-dshm\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920652 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a5c4a0-be20-4561-be9d-956c030894e7-tls-certs\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920770    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-home\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920809    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-model-cache\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920846    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-kserve-provision-location\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.920918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.920897    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-tmp-dir\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.922827 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.922794    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-dshm\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.923135 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.923120    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a5c4a0-be20-4561-be9d-956c030894e7-tls-certs\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:32.933112 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:32.933085    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64sn6\" (UniqueName: \"kubernetes.io/projected/f7a5c4a0-be20-4561-be9d-956c030894e7-kube-api-access-64sn6\") pod \"auth-enabled-test-kserve-566f9cdfc7-k88bd\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:33.016132 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:33.016088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:48:33.048913 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:33.048887    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:33.197803 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:33.197777    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"]
Jul 06 20:48:33.199604 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:48:33.199576    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a5c4a0_be20_4561_be9d_956c030894e7.slice/crio-03d6dec8271aa7b3b7a8f593cbe8d8c15acb7462eebb11ab5427fb1d8bab0e2f WatchSource:0}: Error finding container 03d6dec8271aa7b3b7a8f593cbe8d8c15acb7462eebb11ab5427fb1d8bab0e2f: Status 404 returned error can't find the container with id 03d6dec8271aa7b3b7a8f593cbe8d8c15acb7462eebb11ab5427fb1d8bab0e2f
Jul 06 20:48:33.443110 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:33.443065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" event={"ID":"f7a5c4a0-be20-4561-be9d-956c030894e7","Type":"ContainerStarted","Data":"d613eefddc615b53c70839a8b0796536449cf72a157a98d577220fe6376c341c"}
Jul 06 20:48:33.443283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:33.443131    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" event={"ID":"f7a5c4a0-be20-4561-be9d-956c030894e7","Type":"ContainerStarted","Data":"03d6dec8271aa7b3b7a8f593cbe8d8c15acb7462eebb11ab5427fb1d8bab0e2f"}
Jul 06 20:48:34.259700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:34.259663    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:34.260205 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:34.260195    2582 scope.go:117] "RemoveContainer" containerID="60cc2f8c9c8fc1f646c263943928c0eef23e8a38494e6ae63c6121eb5c3d58ed"
Jul 06 20:48:34.260437 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:48:34.260381    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899"
Jul 06 20:48:34.325866 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:34.325831    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:37.472268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:37.472235    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerID="d613eefddc615b53c70839a8b0796536449cf72a157a98d577220fe6376c341c" exitCode=0
Jul 06 20:48:37.472743 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:37.472317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" event={"ID":"f7a5c4a0-be20-4561-be9d-956c030894e7","Type":"ContainerDied","Data":"d613eefddc615b53c70839a8b0796536449cf72a157a98d577220fe6376c341c"}
Jul 06 20:48:45.331660 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:45.331391    2582 scope.go:117] "RemoveContainer" containerID="60cc2f8c9c8fc1f646c263943928c0eef23e8a38494e6ae63c6121eb5c3d58ed"
Jul 06 20:48:45.332700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:45.332668    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:48:45.332769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:45.332752    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:48:45.332816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:45.332795    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:48:45.544432 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:45.544401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerStarted","Data":"8501700fe4571549ff53433f47ca4f02cfcf6e29d9da48b7e3b6da7b01f1b2a1"}
Jul 06 20:48:45.768372 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:45.768336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:46.773211 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:46.773179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:47.555613 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:47.555581    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:47.556659 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:47.556630    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:48:55.911516 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:55.911482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:57.844381 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:57.844341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:57.847904 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:57.847877    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"]
Jul 06 20:48:57.859388 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:57.859356    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"]
Jul 06 20:48:57.859668 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:57.859639    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="tokenizer" containerID="cri-o://e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef" gracePeriod=60
Jul 06 20:48:57.872370 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:48:57.872331    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:48:57.872370 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:48:57.872370 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:48:57.872370 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1" cmd=["/bin/sleep","15"]
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:57.872397    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" message="" err=<
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:48:57.872416    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main" containerID="cri-o://67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1"
Jul 06 20:48:57.872679 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:57.872461    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main" containerID="cri-o://67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1" gracePeriod=60
Jul 06 20:48:57.926375 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:57.926343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:48:58.632318 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:58.632289    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9_d908d052-8197-473b-bbd8-e5e9b8b3d7a7/main/0.log"
Jul 06 20:48:58.632820 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:58.632784    2582 generic.go:356] "Generic (PLEG): container finished" podID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerID="67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1" exitCode=0
Jul 06 20:48:58.632926 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:58.632863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerDied","Data":"67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1"}
Jul 06 20:48:58.632996 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:58.632926    2582 scope.go:117] "RemoveContainer" containerID="aed886fdda0e04f6d783e8450b2c69e71ee88456d4adbcefe6d71a5e4e03d0d8"
Jul 06 20:48:59.520208 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.520180    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:59.610683 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.610595    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kserve-provision-location\") pod \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") "
Jul 06 20:48:59.610683 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.610672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tls-certs\") pod \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") "
Jul 06 20:48:59.610917 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.610699    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwkqw\" (UniqueName: \"kubernetes.io/projected/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kube-api-access-wwkqw\") pod \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") "
Jul 06 20:48:59.610917 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.610736    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-tmp\") pod \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") "
Jul 06 20:48:59.610917 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.610853    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-cache\") pod \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") "
Jul 06 20:48:59.611131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.610949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-uds\") pod \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\" (UID: \"d908d052-8197-473b-bbd8-e5e9b8b3d7a7\") "
Jul 06 20:48:59.611346 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.611314    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d908d052-8197-473b-bbd8-e5e9b8b3d7a7" (UID: "d908d052-8197-473b-bbd8-e5e9b8b3d7a7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:59.611346 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.611337    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d908d052-8197-473b-bbd8-e5e9b8b3d7a7" (UID: "d908d052-8197-473b-bbd8-e5e9b8b3d7a7"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:59.611551 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.611492    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d908d052-8197-473b-bbd8-e5e9b8b3d7a7" (UID: "d908d052-8197-473b-bbd8-e5e9b8b3d7a7"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:59.611775 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.611745    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d908d052-8197-473b-bbd8-e5e9b8b3d7a7" (UID: "d908d052-8197-473b-bbd8-e5e9b8b3d7a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:59.613268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.613244    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d908d052-8197-473b-bbd8-e5e9b8b3d7a7" (UID: "d908d052-8197-473b-bbd8-e5e9b8b3d7a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:48:59.613268 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.613259    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kube-api-access-wwkqw" (OuterVolumeSpecName: "kube-api-access-wwkqw") pod "d908d052-8197-473b-bbd8-e5e9b8b3d7a7" (UID: "d908d052-8197-473b-bbd8-e5e9b8b3d7a7"). InnerVolumeSpecName "kube-api-access-wwkqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:48:59.641558 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.641526    2582 generic.go:356] "Generic (PLEG): container finished" podID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerID="e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef" exitCode=0
Jul 06 20:48:59.641706 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.641594    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"
Jul 06 20:48:59.641706 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.641596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerDied","Data":"e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef"}
Jul 06 20:48:59.641816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.641749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9" event={"ID":"d908d052-8197-473b-bbd8-e5e9b8b3d7a7","Type":"ContainerDied","Data":"4f3f9df12cbd1804546fa84c08bee4bd0a318c871bef77dc05619b6328ec279f"}
Jul 06 20:48:59.641816 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.641781    2582 scope.go:117] "RemoveContainer" containerID="67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1"
Jul 06 20:48:59.653188 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.653166    2582 scope.go:117] "RemoveContainer" containerID="e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef"
Jul 06 20:48:59.665919 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.665894    2582 scope.go:117] "RemoveContainer" containerID="04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735"
Jul 06 20:48:59.670495 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.670464    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"]
Jul 06 20:48:59.674822 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.674804    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9"]
Jul 06 20:48:59.677523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.677501    2582 scope.go:117] "RemoveContainer" containerID="67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1"
Jul 06 20:48:59.677808 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:48:59.677786    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1\": container with ID starting with 67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1 not found: ID does not exist" containerID="67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1"
Jul 06 20:48:59.677890 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.677817    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1"} err="failed to get container status \"67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1\": rpc error: code = NotFound desc = could not find container \"67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1\": container with ID starting with 67ba5f5650da25a5db11bd9b7fb442d348d74155b372455facde38ea733edda1 not found: ID does not exist"
Jul 06 20:48:59.677890 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.677837    2582 scope.go:117] "RemoveContainer" containerID="e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef"
Jul 06 20:48:59.678134 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:48:59.678115    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef\": container with ID starting with e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef not found: ID does not exist" containerID="e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef"
Jul 06 20:48:59.678177 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.678143    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef"} err="failed to get container status \"e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef\": rpc error: code = NotFound desc = could not find container \"e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef\": container with ID starting with e8e332c1acae7e23eddc67a8340b725b87b029842a9afa97ce7ad21c2222b7ef not found: ID does not exist"
Jul 06 20:48:59.678177 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.678161    2582 scope.go:117] "RemoveContainer" containerID="04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735"
Jul 06 20:48:59.678431 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:48:59.678408    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735\": container with ID starting with 04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735 not found: ID does not exist" containerID="04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735"
Jul 06 20:48:59.678521 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.678437    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735"} err="failed to get container status \"04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735\": rpc error: code = NotFound desc = could not find container \"04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735\": container with ID starting with 04fd2f8c003086a9fd3f61b2fa41e6c6a858fbbdec7b762096e4100f48da9735 not found: ID does not exist"
Jul 06 20:48:59.711936 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.711898    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:48:59.711936 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.711928    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwkqw\" (UniqueName: \"kubernetes.io/projected/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kube-api-access-wwkqw\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:48:59.711936 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.711941    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-tmp\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:48:59.712169 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.711954    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:48:59.712169 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.711981    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:48:59.712169 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:48:59.711994    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d908d052-8197-473b-bbd8-e5e9b8b3d7a7-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:01.336958 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:01.336926    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" path="/var/lib/kubelet/pods/d908d052-8197-473b-bbd8-e5e9b8b3d7a7/volumes"
Jul 06 20:49:12.878354 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:12.878314    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" containerID="cri-o://ceeeddbe1a849cfe2ffabbb52196be8e7ac89d62f18baf839ec77c978ebcaaaa" gracePeriod=45
Jul 06 20:49:13.015951 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:13.015914    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 20:49:14.015990 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:14.015886    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 20:49:14.016466 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:14.016261    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:49:15.016570 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:15.016521    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 20:49:16.016073 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:16.016026    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 20:49:17.015920 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:17.015873    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 20:49:18.016245 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:18.016198    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 06 20:49:18.761155 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:18.761120    2582 generic.go:356] "Generic (PLEG): container finished" podID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerID="ceeeddbe1a849cfe2ffabbb52196be8e7ac89d62f18baf839ec77c978ebcaaaa" exitCode=0
Jul 06 20:49:18.761335 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:18.761196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" event={"ID":"7bf81f5b-d1ea-42e2-ad83-2534d60529d4","Type":"ContainerDied","Data":"ceeeddbe1a849cfe2ffabbb52196be8e7ac89d62f18baf839ec77c978ebcaaaa"}
Jul 06 20:49:20.016726 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.016671    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 06 20:49:20.190125 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.190094    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:49:20.222335 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222306    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-home\") pod \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") "
Jul 06 20:49:20.222335 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-dshm\") pod \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") "
Jul 06 20:49:20.222532 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222419    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tls-certs\") pod \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") "
Jul 06 20:49:20.222532 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tmp-dir\") pod \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") "
Jul 06 20:49:20.222532 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222522    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-model-cache\") pod \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") "
Jul 06 20:49:20.222685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2czx\" (UniqueName: \"kubernetes.io/projected/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kube-api-access-k2czx\") pod \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") "
Jul 06 20:49:20.222685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kserve-provision-location\") pod \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\" (UID: \"7bf81f5b-d1ea-42e2-ad83-2534d60529d4\") "
Jul 06 20:49:20.222685 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222584    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-home" (OuterVolumeSpecName: "home") pod "7bf81f5b-d1ea-42e2-ad83-2534d60529d4" (UID: "7bf81f5b-d1ea-42e2-ad83-2534d60529d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:20.222865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222753    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bf81f5b-d1ea-42e2-ad83-2534d60529d4" (UID: "7bf81f5b-d1ea-42e2-ad83-2534d60529d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:20.222865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.222820    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bf81f5b-d1ea-42e2-ad83-2534d60529d4" (UID: "7bf81f5b-d1ea-42e2-ad83-2534d60529d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:20.223045 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.223021    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:20.223183 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.223070    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:20.223183 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.223086    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:20.224668 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.224643    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-dshm" (OuterVolumeSpecName: "dshm") pod "7bf81f5b-d1ea-42e2-ad83-2534d60529d4" (UID: "7bf81f5b-d1ea-42e2-ad83-2534d60529d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:20.224987 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.224945    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bf81f5b-d1ea-42e2-ad83-2534d60529d4" (UID: "7bf81f5b-d1ea-42e2-ad83-2534d60529d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:49:20.224987 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.224956    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kube-api-access-k2czx" (OuterVolumeSpecName: "kube-api-access-k2czx") pod "7bf81f5b-d1ea-42e2-ad83-2534d60529d4" (UID: "7bf81f5b-d1ea-42e2-ad83-2534d60529d4"). InnerVolumeSpecName "kube-api-access-k2czx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:49:20.290347 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.290267    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bf81f5b-d1ea-42e2-ad83-2534d60529d4" (UID: "7bf81f5b-d1ea-42e2-ad83-2534d60529d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:49:20.323529 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.323498    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2czx\" (UniqueName: \"kubernetes.io/projected/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kube-api-access-k2czx\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:20.323529 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.323525    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:20.323710 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.323536    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:20.323710 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.323547    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf81f5b-d1ea-42e2-ad83-2534d60529d4-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:49:20.779884 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.779849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" event={"ID":"7bf81f5b-d1ea-42e2-ad83-2534d60529d4","Type":"ContainerDied","Data":"5c243621f22ba28a5320528ce0872822afc60055568c1c4a1939c6e87bcef0b0"}
Jul 06 20:49:20.779884 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.779870    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"
Jul 06 20:49:20.780159 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.779924    2582 scope.go:117] "RemoveContainer" containerID="ceeeddbe1a849cfe2ffabbb52196be8e7ac89d62f18baf839ec77c978ebcaaaa"
Jul 06 20:49:20.781884 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.781858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" event={"ID":"f7a5c4a0-be20-4561-be9d-956c030894e7","Type":"ContainerStarted","Data":"b8222f80fbc398e223f3e763eb1cf0dafd846ee4704be6f2fe3c625a3cb7bb9c"}
Jul 06 20:49:20.789756 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.789731    2582 scope.go:117] "RemoveContainer" containerID="19f882878c1f33e903f8ca09849e17844f61cdecf7102bb5b18f1ca70f0b3c90"
Jul 06 20:49:20.815119 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.815036    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podStartSLOduration=2.221495515 podStartE2EDuration="48.815016957s" totalImagesPullingTime="43.086311373s" totalInitContainerRuntime="3.507210069s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:20.811288248 +0000 UTC m=+858.145628843" watchObservedRunningTime="2026-07-06 20:49:20.815016957 +0000 UTC m=+858.149357648"
Jul 06 20:49:20.829355 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.829326    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"]
Jul 06 20:49:20.836052 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:20.836025    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4"]
Jul 06 20:49:21.016447 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:21.016403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:49:21.017493 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:21.017462    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 06 20:49:21.337144 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:21.337111    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" path="/var/lib/kubelet/pods/7bf81f5b-d1ea-42e2-ad83-2534d60529d4/volumes"
Jul 06 20:49:23.017303 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:23.017269    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:49:23.018751 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:23.018717    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:49:28.698562 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.698524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:28.916091 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916054    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"]
Jul 06 20:49:28.916635 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916602    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main"
Jul 06 20:49:28.916635 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916635    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916653    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="tokenizer"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916662    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="tokenizer"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916698    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="storage-initializer"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916708    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="storage-initializer"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916724    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="storage-initializer"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916732    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="storage-initializer"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916738    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916746    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916754    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main"
Jul 06 20:49:28.916812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916759    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main"
Jul 06 20:49:28.917338 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916873    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="tokenizer"
Jul 06 20:49:28.917338 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916882    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main"
Jul 06 20:49:28.917338 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.916889    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf81f5b-d1ea-42e2-ad83-2534d60529d4" containerName="main"
Jul 06 20:49:28.937528 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.937498    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"]
Jul 06 20:49:28.937680 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.937644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:28.941299 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.941277    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-vq64c\""
Jul 06 20:49:28.942646 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.942619    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:28.943314 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:28.943287    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 06 20:49:29.005761 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.005667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.005938 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.005770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbqn\" (UniqueName: \"kubernetes.io/projected/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-kube-api-access-rfbqn\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.005938 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.005807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.107229 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.107193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.107405 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.107266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbqn\" (UniqueName: \"kubernetes.io/projected/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-kube-api-access-rfbqn\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.107405 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.107308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.107703 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.107676    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.109748 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.109718    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.116589 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.116562    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbqn\" (UniqueName: \"kubernetes.io/projected/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-kube-api-access-rfbqn\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.248682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.248644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.416844 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.416817    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"]
Jul 06 20:49:29.418387 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:49:29.418355    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16a3b94_39b0_47b7_869f_50fbba7bb5bf.slice/crio-7b540f8639b30ab2a0d78f0e8df7f65b7bdee0f30196092a4a1c6d82071eed74 WatchSource:0}: Error finding container 7b540f8639b30ab2a0d78f0e8df7f65b7bdee0f30196092a4a1c6d82071eed74: Status 404 returned error can't find the container with id 7b540f8639b30ab2a0d78f0e8df7f65b7bdee0f30196092a4a1c6d82071eed74
Jul 06 20:49:29.839535 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.839407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" event={"ID":"a16a3b94-39b0-47b7-869f-50fbba7bb5bf","Type":"ContainerStarted","Data":"bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6"}
Jul 06 20:49:29.840023 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.839575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:49:29.840023 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.839619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" event={"ID":"a16a3b94-39b0-47b7-869f-50fbba7bb5bf","Type":"ContainerStarted","Data":"7b540f8639b30ab2a0d78f0e8df7f65b7bdee0f30196092a4a1c6d82071eed74"}
Jul 06 20:49:29.863392 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:29.863341    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" podStartSLOduration=1.863323965 podStartE2EDuration="1.863323965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:29.862467003 +0000 UTC m=+867.196807600" watchObservedRunningTime="2026-07-06 20:49:29.863323965 +0000 UTC m=+867.197664560"
Jul 06 20:49:33.017226 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:33.017180    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:49:43.017620 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:43.017513    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:49:45.821408 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:45.821348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:53.017479 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:49:53.017429    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:50:00.866355 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:00.866269    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:50:02.243711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.243678    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"]
Jul 06 20:50:02.253908 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:50:02.253873    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:50:02.253908 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:50:02.253908 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:50:02.253908 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6" cmd=["/bin/sleep","15"]
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.253930    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" message="" err=<
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:50:02.253948    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" podUID="a16a3b94-39b0-47b7-869f-50fbba7bb5bf" containerName="main" containerID="cri-o://bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6"
Jul 06 20:50:02.254250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.254004    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" podUID="a16a3b94-39b0-47b7-869f-50fbba7bb5bf" containerName="main" containerID="cri-o://bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6" gracePeriod=60
Jul 06 20:50:02.304981 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.304921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:02.378596 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.378561    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:02.515376 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.515353    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:50:02.528427 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.528390    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfbqn\" (UniqueName: \"kubernetes.io/projected/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-kube-api-access-rfbqn\") pod \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") "
Jul 06 20:50:02.528554 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.528448    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tokenizer-uds\") pod \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") "
Jul 06 20:50:02.528629 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.528580    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tls-certs\") pod \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\" (UID: \"a16a3b94-39b0-47b7-869f-50fbba7bb5bf\") "
Jul 06 20:50:02.528710 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.528666    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a16a3b94-39b0-47b7-869f-50fbba7bb5bf" (UID: "a16a3b94-39b0-47b7-869f-50fbba7bb5bf"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:02.528864 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.528847    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:50:02.530564 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.530534    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-kube-api-access-rfbqn" (OuterVolumeSpecName: "kube-api-access-rfbqn") pod "a16a3b94-39b0-47b7-869f-50fbba7bb5bf" (UID: "a16a3b94-39b0-47b7-869f-50fbba7bb5bf"). InnerVolumeSpecName "kube-api-access-rfbqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:02.530663 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.530593    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a16a3b94-39b0-47b7-869f-50fbba7bb5bf" (UID: "a16a3b94-39b0-47b7-869f-50fbba7bb5bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:50:02.629951 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.629915    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:50:02.629951 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:02.629945    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfbqn\" (UniqueName: \"kubernetes.io/projected/a16a3b94-39b0-47b7-869f-50fbba7bb5bf-kube-api-access-rfbqn\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:50:03.017198 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.017151    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:50:03.057536 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.057501    2582 generic.go:356] "Generic (PLEG): container finished" podID="a16a3b94-39b0-47b7-869f-50fbba7bb5bf" containerID="bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6" exitCode=0
Jul 06 20:50:03.057698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.057561    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"
Jul 06 20:50:03.057698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.057581    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" event={"ID":"a16a3b94-39b0-47b7-869f-50fbba7bb5bf","Type":"ContainerDied","Data":"bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6"}
Jul 06 20:50:03.057698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.057658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk" event={"ID":"a16a3b94-39b0-47b7-869f-50fbba7bb5bf","Type":"ContainerDied","Data":"7b540f8639b30ab2a0d78f0e8df7f65b7bdee0f30196092a4a1c6d82071eed74"}
Jul 06 20:50:03.057698 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.057679    2582 scope.go:117] "RemoveContainer" containerID="bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6"
Jul 06 20:50:03.067799 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.067773    2582 scope.go:117] "RemoveContainer" containerID="bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6"
Jul 06 20:50:03.068144 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:50:03.068119    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6\": container with ID starting with bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6 not found: ID does not exist" containerID="bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6"
Jul 06 20:50:03.068226 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.068185    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6"} err="failed to get container status \"bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6\": rpc error: code = NotFound desc = could not find container \"bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6\": container with ID starting with bdf395ee5ad2834c6845fbffc703ff7c57d9aeb3366bf0249daffe959e3b5bc6 not found: ID does not exist"
Jul 06 20:50:03.082526 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.082499    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"]
Jul 06 20:50:03.088801 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.088768    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk"]
Jul 06 20:50:03.158481 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.158454    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:50:03.168192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.168168    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:50:03.339083 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:03.339009    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a16a3b94-39b0-47b7-869f-50fbba7bb5bf" path="/var/lib/kubelet/pods/a16a3b94-39b0-47b7-869f-50fbba7bb5bf/volumes"
Jul 06 20:50:13.017643 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:13.017593    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:50:23.017459 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:23.017414    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:50:33.017244 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:33.017196    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:50:34.279923 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.279887    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:34.535654 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.535548    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"]
Jul 06 20:50:34.536260 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.536239    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a16a3b94-39b0-47b7-869f-50fbba7bb5bf" containerName="main"
Jul 06 20:50:34.536260 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.536260    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a16a3b94-39b0-47b7-869f-50fbba7bb5bf" containerName="main"
Jul 06 20:50:34.536423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.536355    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a16a3b94-39b0-47b7-869f-50fbba7bb5bf" containerName="main"
Jul 06 20:50:34.536423 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.536368    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d908d052-8197-473b-bbd8-e5e9b8b3d7a7" containerName="main"
Jul 06 20:50:34.538918 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.538894    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"]
Jul 06 20:50:34.539054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.539018    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.541895 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.541864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d799407-1164-4be0-943e-6d1f73e4612a-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.542033 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.541996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7h2\" (UniqueName: \"kubernetes.io/projected/8d799407-1164-4be0-943e-6d1f73e4612a-kube-api-access-4m7h2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.542098 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.542063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8d799407-1164-4be0-943e-6d1f73e4612a-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.543122 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.543099    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 06 20:50:34.543269 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.543098    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-g7ts7\""
Jul 06 20:50:34.544172 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.544151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:34.642523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.642487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8d799407-1164-4be0-943e-6d1f73e4612a-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.642717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.642550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d799407-1164-4be0-943e-6d1f73e4612a-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.642717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.642615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7h2\" (UniqueName: \"kubernetes.io/projected/8d799407-1164-4be0-943e-6d1f73e4612a-kube-api-access-4m7h2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.643016 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.642991    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8d799407-1164-4be0-943e-6d1f73e4612a-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.645183 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.645163    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d799407-1164-4be0-943e-6d1f73e4612a-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.651477 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.651458    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7h2\" (UniqueName: \"kubernetes.io/projected/8d799407-1164-4be0-943e-6d1f73e4612a-kube-api-access-4m7h2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:34.849819 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:34.849712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:35.000625 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:50:35.000596    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d799407_1164_4be0_943e_6d1f73e4612a.slice/crio-d86c6bb94f9846a168b03c3ece1b993f6d85a2ab05fedfe29febd4ea933caadf WatchSource:0}: Error finding container d86c6bb94f9846a168b03c3ece1b993f6d85a2ab05fedfe29febd4ea933caadf: Status 404 returned error can't find the container with id d86c6bb94f9846a168b03c3ece1b993f6d85a2ab05fedfe29febd4ea933caadf
Jul 06 20:50:35.000775 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:35.000651    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"]
Jul 06 20:50:35.253606 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:35.253566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" event={"ID":"8d799407-1164-4be0-943e-6d1f73e4612a","Type":"ContainerStarted","Data":"11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344"}
Jul 06 20:50:35.253814 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:35.253629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" event={"ID":"8d799407-1164-4be0-943e-6d1f73e4612a","Type":"ContainerStarted","Data":"d86c6bb94f9846a168b03c3ece1b993f6d85a2ab05fedfe29febd4ea933caadf"}
Jul 06 20:50:35.253814 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:35.253667    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:50:35.280096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:35.280037    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" podStartSLOduration=1.280015715 podStartE2EDuration="1.280015715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:35.278869365 +0000 UTC m=+932.613209961" watchObservedRunningTime="2026-07-06 20:50:35.280015715 +0000 UTC m=+932.614356311"
Jul 06 20:50:40.081272 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:40.081240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:50:43.017128 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:43.017038    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:50:53.017361 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:53.017319    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:50:55.419853 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:50:55.419802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:03.017158 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:03.017113    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:51:06.260668 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:06.260635    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:51:13.026812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:13.026778    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:51:13.034906 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:13.034879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:51:17.924675 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:17.924635    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:17.946605 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:17.946573    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"]
Jul 06 20:51:18.017575 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:18.017543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:32.977091 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:32.976985    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" containerID="cri-o://b8222f80fbc398e223f3e763eb1cf0dafd846ee4704be6f2fe3c625a3cb7bb9c" gracePeriod=45
Jul 06 20:51:33.027852 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.027809    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 06 20:51:33.594474 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.594437    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerID="b8222f80fbc398e223f3e763eb1cf0dafd846ee4704be6f2fe3c625a3cb7bb9c" exitCode=0
Jul 06 20:51:33.594570 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.594509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" event={"ID":"f7a5c4a0-be20-4561-be9d-956c030894e7","Type":"ContainerDied","Data":"b8222f80fbc398e223f3e763eb1cf0dafd846ee4704be6f2fe3c625a3cb7bb9c"}
Jul 06 20:51:33.729909 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.729885    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:51:33.831452 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831371    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-dshm\") pod \"f7a5c4a0-be20-4561-be9d-956c030894e7\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") "
Jul 06 20:51:33.831452 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-model-cache\") pod \"f7a5c4a0-be20-4561-be9d-956c030894e7\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") "
Jul 06 20:51:33.831452 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-home\") pod \"f7a5c4a0-be20-4561-be9d-956c030894e7\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") "
Jul 06 20:51:33.831730 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64sn6\" (UniqueName: \"kubernetes.io/projected/f7a5c4a0-be20-4561-be9d-956c030894e7-kube-api-access-64sn6\") pod \"f7a5c4a0-be20-4561-be9d-956c030894e7\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") "
Jul 06 20:51:33.831730 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831509    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-kserve-provision-location\") pod \"f7a5c4a0-be20-4561-be9d-956c030894e7\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") "
Jul 06 20:51:33.831730 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a5c4a0-be20-4561-be9d-956c030894e7-tls-certs\") pod \"f7a5c4a0-be20-4561-be9d-956c030894e7\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") "
Jul 06 20:51:33.831730 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831600    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-tmp-dir\") pod \"f7a5c4a0-be20-4561-be9d-956c030894e7\" (UID: \"f7a5c4a0-be20-4561-be9d-956c030894e7\") "
Jul 06 20:51:33.831730 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.831698    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7a5c4a0-be20-4561-be9d-956c030894e7" (UID: "f7a5c4a0-be20-4561-be9d-956c030894e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:51:33.832041 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.832013    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:51:33.832259 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.832228    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-home" (OuterVolumeSpecName: "home") pod "f7a5c4a0-be20-4561-be9d-956c030894e7" (UID: "f7a5c4a0-be20-4561-be9d-956c030894e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:51:33.833718 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.833687    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a5c4a0-be20-4561-be9d-956c030894e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7a5c4a0-be20-4561-be9d-956c030894e7" (UID: "f7a5c4a0-be20-4561-be9d-956c030894e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:51:33.833718 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.833706    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a5c4a0-be20-4561-be9d-956c030894e7-kube-api-access-64sn6" (OuterVolumeSpecName: "kube-api-access-64sn6") pod "f7a5c4a0-be20-4561-be9d-956c030894e7" (UID: "f7a5c4a0-be20-4561-be9d-956c030894e7"). InnerVolumeSpecName "kube-api-access-64sn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:51:33.833837 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.833756    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-dshm" (OuterVolumeSpecName: "dshm") pod "f7a5c4a0-be20-4561-be9d-956c030894e7" (UID: "f7a5c4a0-be20-4561-be9d-956c030894e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:51:33.847130 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.847099    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7a5c4a0-be20-4561-be9d-956c030894e7" (UID: "f7a5c4a0-be20-4561-be9d-956c030894e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:51:33.889658 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.889621    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7a5c4a0-be20-4561-be9d-956c030894e7" (UID: "f7a5c4a0-be20-4561-be9d-956c030894e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:51:33.932978 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.932940    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a5c4a0-be20-4561-be9d-956c030894e7-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:51:33.933123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.932984    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:51:33.933123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.932993    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:51:33.933123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.933001    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:51:33.933123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.933011    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64sn6\" (UniqueName: \"kubernetes.io/projected/f7a5c4a0-be20-4561-be9d-956c030894e7-kube-api-access-64sn6\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:51:33.933123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:33.933021    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a5c4a0-be20-4561-be9d-956c030894e7-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:51:34.602227 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:34.602197    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"
Jul 06 20:51:34.602227 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:34.602218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd" event={"ID":"f7a5c4a0-be20-4561-be9d-956c030894e7","Type":"ContainerDied","Data":"03d6dec8271aa7b3b7a8f593cbe8d8c15acb7462eebb11ab5427fb1d8bab0e2f"}
Jul 06 20:51:34.602769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:34.602271    2582 scope.go:117] "RemoveContainer" containerID="b8222f80fbc398e223f3e763eb1cf0dafd846ee4704be6f2fe3c625a3cb7bb9c"
Jul 06 20:51:34.612128 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:34.612112    2582 scope.go:117] "RemoveContainer" containerID="d613eefddc615b53c70839a8b0796536449cf72a157a98d577220fe6376c341c"
Jul 06 20:51:34.626608 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:34.626585    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"]
Jul 06 20:51:34.632173 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:34.632149    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd"]
Jul 06 20:51:35.337446 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:35.337414    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" path="/var/lib/kubelet/pods/f7a5c4a0-be20-4561-be9d-956c030894e7/volumes"
Jul 06 20:51:50.192737 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.192693    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"]
Jul 06 20:51:50.193212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.193145    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="storage-initializer"
Jul 06 20:51:50.193212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.193160    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="storage-initializer"
Jul 06 20:51:50.193212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.193183    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main"
Jul 06 20:51:50.193212 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.193189    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main"
Jul 06 20:51:50.193344 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.193263    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a5c4a0-be20-4561-be9d-956c030894e7" containerName="main"
Jul 06 20:51:50.196499 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.196481    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.200719 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.200696    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 06 20:51:50.219055 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.219023    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"]
Jul 06 20:51:50.269335 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.269301    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:50.280406 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.280375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.280553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.280411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b692bc0-7add-4910-a45c-a62dd42a5757-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.280553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.280432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.280553 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.280510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gbr\" (UniqueName: \"kubernetes.io/projected/3b692bc0-7add-4910-a45c-a62dd42a5757-kube-api-access-g4gbr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.280704 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.280563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.280704 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.280660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.381476 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.381445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gbr\" (UniqueName: \"kubernetes.io/projected/3b692bc0-7add-4910-a45c-a62dd42a5757-kube-api-access-g4gbr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.381691 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.381640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.381823 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.381799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.381957 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.381938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.382058 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.381998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b692bc0-7add-4910-a45c-a62dd42a5757-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.382058 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.382038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.382163 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.382096    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.382163 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.382141    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.382479 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.382455    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.384222 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.384195    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.384477 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.384460    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b692bc0-7add-4910-a45c-a62dd42a5757-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.397868 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.397843    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gbr\" (UniqueName: \"kubernetes.io/projected/3b692bc0-7add-4910-a45c-a62dd42a5757-kube-api-access-g4gbr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.507471 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.507394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:51:50.679811 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.679778    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"]
Jul 06 20:51:50.681409 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:51:50.681382    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b692bc0_7add_4910_a45c_a62dd42a5757.slice/crio-9abcaedb3a2adb90a6035fbe74f8f9c55964aaccd4ef74ae88feb0777d9c4f37 WatchSource:0}: Error finding container 9abcaedb3a2adb90a6035fbe74f8f9c55964aaccd4ef74ae88feb0777d9c4f37: Status 404 returned error can't find the container with id 9abcaedb3a2adb90a6035fbe74f8f9c55964aaccd4ef74ae88feb0777d9c4f37
Jul 06 20:51:50.690975 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:50.690939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d" event={"ID":"3b692bc0-7add-4910-a45c-a62dd42a5757","Type":"ContainerStarted","Data":"9abcaedb3a2adb90a6035fbe74f8f9c55964aaccd4ef74ae88feb0777d9c4f37"}
Jul 06 20:51:51.698703 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:51.698654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d" event={"ID":"3b692bc0-7add-4910-a45c-a62dd42a5757","Type":"ContainerStarted","Data":"7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676"}
Jul 06 20:51:51.751747 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:51.751683    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d" podStartSLOduration=1.751667956 podStartE2EDuration="1.751667956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:51.749742221 +0000 UTC m=+1009.084082829" watchObservedRunningTime="2026-07-06 20:51:51.751667956 +0000 UTC m=+1009.086008551"
Jul 06 20:51:51.832727 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:51.832691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:51:52.508525 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:51:52.508489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:52:00.508318 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.508282    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:52:00.520891 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.520860    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:52:00.764898 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.764822    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:52:00.798056 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.798020    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"]
Jul 06 20:52:00.802983 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.802948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.806858 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.806834    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:52:00.807614 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.807594    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 06 20:52:00.829520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.829488    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"]
Jul 06 20:52:00.844908 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.844881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:00.880793 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.880766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-tmp-dir\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.881000 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.880818    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nn5x\" (UniqueName: \"kubernetes.io/projected/52a56f93-9f85-4005-ba7b-195522086641-kube-api-access-8nn5x\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.881000 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.880878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-dshm\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.881000 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.880904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-home\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.881175 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.881029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-model-cache\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.881175 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.881071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.881254 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.881165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a56f93-9f85-4005-ba7b-195522086641-tls-certs\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.982700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.982662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-model-cache\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.982887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.982706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.982887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.982747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a56f93-9f85-4005-ba7b-195522086641-tls-certs\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.982887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.982830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-tmp-dir\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.982887 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.982869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn5x\" (UniqueName: \"kubernetes.io/projected/52a56f93-9f85-4005-ba7b-195522086641-kube-api-access-8nn5x\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.983130 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.983027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-dshm\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.983130 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.983072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-home\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.983240 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.983184    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-model-cache\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.983301 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.983229    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.983301 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.983277    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-tmp-dir\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.983411 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.983398    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-home\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.985274 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.985251    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-dshm\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.985552 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.985532    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a56f93-9f85-4005-ba7b-195522086641-tls-certs\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:00.992302 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:00.992278    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nn5x\" (UniqueName: \"kubernetes.io/projected/52a56f93-9f85-4005-ba7b-195522086641-kube-api-access-8nn5x\") pod \"auth-invalid-token-test-kserve-754cb7b645-9cmz7\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:01.114681 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:01.114593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:01.158435 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:01.158394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:01.285249 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:01.285221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"]
Jul 06 20:52:01.286661 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:52:01.286634    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a56f93_9f85_4005_ba7b_195522086641.slice/crio-ffbe6f9a53b8df26f2e4663c4bb532c0f5da775e696c2b3d79a1713d85ba5e6e WatchSource:0}: Error finding container ffbe6f9a53b8df26f2e4663c4bb532c0f5da775e696c2b3d79a1713d85ba5e6e: Status 404 returned error can't find the container with id ffbe6f9a53b8df26f2e4663c4bb532c0f5da775e696c2b3d79a1713d85ba5e6e
Jul 06 20:52:01.762524 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:01.762488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" event={"ID":"52a56f93-9f85-4005-ba7b-195522086641","Type":"ContainerStarted","Data":"39445e574ff91abcac9d4f39b2c504d6bc1981a24e159d1a50e2ec11424dc582"}
Jul 06 20:52:01.762925 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:01.762540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" event={"ID":"52a56f93-9f85-4005-ba7b-195522086641","Type":"ContainerStarted","Data":"ffbe6f9a53b8df26f2e4663c4bb532c0f5da775e696c2b3d79a1713d85ba5e6e"}
Jul 06 20:52:05.786896 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:05.786861    2582 generic.go:356] "Generic (PLEG): container finished" podID="52a56f93-9f85-4005-ba7b-195522086641" containerID="39445e574ff91abcac9d4f39b2c504d6bc1981a24e159d1a50e2ec11424dc582" exitCode=0
Jul 06 20:52:05.786896 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:05.786890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" event={"ID":"52a56f93-9f85-4005-ba7b-195522086641","Type":"ContainerDied","Data":"39445e574ff91abcac9d4f39b2c504d6bc1981a24e159d1a50e2ec11424dc582"}
Jul 06 20:52:06.795133 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:06.795099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" event={"ID":"52a56f93-9f85-4005-ba7b-195522086641","Type":"ContainerStarted","Data":"f6cba471fa29284f0fe8cbf3a07207b248d7bf80f2fbf84afd1f0128cbcc05c4"}
Jul 06 20:52:06.827126 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:06.827078    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podStartSLOduration=3.005808724 podStartE2EDuration="6.827062487s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.821253763s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:52:06.822885495 +0000 UTC m=+1024.157226109" watchObservedRunningTime="2026-07-06 20:52:06.827062487 +0000 UTC m=+1024.161403082"
Jul 06 20:52:07.115426 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:07.115316    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:11.115276 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:11.115243    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:52:11.117151 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:11.117115    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:52:21.115543 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:21.115495    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:52:31.115420 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:31.115370    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:52:39.446794 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:39.446758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:41.115920 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:41.115873    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:52:51.115454 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:51.115401    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:52:53.610023 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:53.609988    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:55.689271 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:55.689168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:52:55.713223 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:55.713178    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"]
Jul 06 20:52:55.774579 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:52:55.774529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:01.116016 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:01.115958    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:53:10.733900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:10.733855    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d" podUID="3b692bc0-7add-4910-a45c-a62dd42a5757" containerName="main" containerID="cri-o://7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676" gracePeriod=45
Jul 06 20:53:10.754247 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:10.754216    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d" podUID="3b692bc0-7add-4910-a45c-a62dd42a5757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 06 20:53:10.991651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:10.991587    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:53:11.078306 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078275    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b692bc0-7add-4910-a45c-a62dd42a5757-tls-certs\") pod \"3b692bc0-7add-4910-a45c-a62dd42a5757\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") "
Jul 06 20:53:11.078504 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-home\") pod \"3b692bc0-7add-4910-a45c-a62dd42a5757\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") "
Jul 06 20:53:11.078504 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-model-cache\") pod \"3b692bc0-7add-4910-a45c-a62dd42a5757\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") "
Jul 06 20:53:11.078504 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078473    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-tmp-dir\") pod \"3b692bc0-7add-4910-a45c-a62dd42a5757\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") "
Jul 06 20:53:11.078504 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-dshm\") pod \"3b692bc0-7add-4910-a45c-a62dd42a5757\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") "
Jul 06 20:53:11.078727 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078548    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4gbr\" (UniqueName: \"kubernetes.io/projected/3b692bc0-7add-4910-a45c-a62dd42a5757-kube-api-access-g4gbr\") pod \"3b692bc0-7add-4910-a45c-a62dd42a5757\" (UID: \"3b692bc0-7add-4910-a45c-a62dd42a5757\") "
Jul 06 20:53:11.078782 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078727    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-home" (OuterVolumeSpecName: "home") pod "3b692bc0-7add-4910-a45c-a62dd42a5757" (UID: "3b692bc0-7add-4910-a45c-a62dd42a5757"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:11.078782 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078749    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b692bc0-7add-4910-a45c-a62dd42a5757" (UID: "3b692bc0-7add-4910-a45c-a62dd42a5757"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:11.078782 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078732    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b692bc0-7add-4910-a45c-a62dd42a5757" (UID: "3b692bc0-7add-4910-a45c-a62dd42a5757"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:11.078959 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078939    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:11.079047 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.078984    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:11.079047 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.079001    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:11.080674 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.080650    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b692bc0-7add-4910-a45c-a62dd42a5757-kube-api-access-g4gbr" (OuterVolumeSpecName: "kube-api-access-g4gbr") pod "3b692bc0-7add-4910-a45c-a62dd42a5757" (UID: "3b692bc0-7add-4910-a45c-a62dd42a5757"). InnerVolumeSpecName "kube-api-access-g4gbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:53:11.081105 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.081082    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b692bc0-7add-4910-a45c-a62dd42a5757-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b692bc0-7add-4910-a45c-a62dd42a5757" (UID: "3b692bc0-7add-4910-a45c-a62dd42a5757"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:53:11.081105 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.081093    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-dshm" (OuterVolumeSpecName: "dshm") pod "3b692bc0-7add-4910-a45c-a62dd42a5757" (UID: "3b692bc0-7add-4910-a45c-a62dd42a5757"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:11.115612 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.115564    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:53:11.179484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.179449    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gbr\" (UniqueName: \"kubernetes.io/projected/3b692bc0-7add-4910-a45c-a62dd42a5757-kube-api-access-g4gbr\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:11.179484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.179482    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b692bc0-7add-4910-a45c-a62dd42a5757-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:11.179484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.179493    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b692bc0-7add-4910-a45c-a62dd42a5757-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:11.194541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.194508    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b692bc0-7add-4910-a45c-a62dd42a5757" containerID="7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676" exitCode=0
Jul 06 20:53:11.194696 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.194574    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"
Jul 06 20:53:11.194696 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.194598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d" event={"ID":"3b692bc0-7add-4910-a45c-a62dd42a5757","Type":"ContainerDied","Data":"7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676"}
Jul 06 20:53:11.194696 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.194650    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d" event={"ID":"3b692bc0-7add-4910-a45c-a62dd42a5757","Type":"ContainerDied","Data":"9abcaedb3a2adb90a6035fbe74f8f9c55964aaccd4ef74ae88feb0777d9c4f37"}
Jul 06 20:53:11.194696 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.194673    2582 scope.go:117] "RemoveContainer" containerID="7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676"
Jul 06 20:53:11.204280 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.204265    2582 scope.go:117] "RemoveContainer" containerID="7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676"
Jul 06 20:53:11.204558 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:53:11.204540    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676\": container with ID starting with 7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676 not found: ID does not exist" containerID="7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676"
Jul 06 20:53:11.204622 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.204567    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676"} err="failed to get container status \"7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676\": rpc error: code = NotFound desc = could not find container \"7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676\": container with ID starting with 7c4fc4a627f6e85fcc5dbaa02fd3390d0f2a09c7993bedbf0e2d6203ddbda676 not found: ID does not exist"
Jul 06 20:53:11.222858 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.222832    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"]
Jul 06 20:53:11.226063 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.226040    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d"]
Jul 06 20:53:11.337471 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:11.337390    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b692bc0-7add-4910-a45c-a62dd42a5757" path="/var/lib/kubelet/pods/3b692bc0-7add-4910-a45c-a62dd42a5757/volumes"
Jul 06 20:53:12.805769 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:12.805737    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"]
Jul 06 20:53:12.809911 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:12.809868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:12.815989 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:53:12.815940    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:53:12.815989 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:53:12.815989 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:53:12.815989 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344" cmd=["/bin/sleep","15"]
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:12.816010    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" message="" err=<
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:53:12.816028    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" podUID="8d799407-1164-4be0-943e-6d1f73e4612a" containerName="main" containerID="cri-o://11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344"
Jul 06 20:53:12.816280 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:12.816073    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" podUID="8d799407-1164-4be0-943e-6d1f73e4612a" containerName="main" containerID="cri-o://11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344" gracePeriod=60
Jul 06 20:53:12.876501 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:12.876474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:13.076972 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.076929    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:53:13.203386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.203352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m7h2\" (UniqueName: \"kubernetes.io/projected/8d799407-1164-4be0-943e-6d1f73e4612a-kube-api-access-4m7h2\") pod \"8d799407-1164-4be0-943e-6d1f73e4612a\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") "
Jul 06 20:53:13.203386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.203395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8d799407-1164-4be0-943e-6d1f73e4612a-tokenizer-uds\") pod \"8d799407-1164-4be0-943e-6d1f73e4612a\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") "
Jul 06 20:53:13.203641 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.203441    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d799407-1164-4be0-943e-6d1f73e4612a-tls-certs\") pod \"8d799407-1164-4be0-943e-6d1f73e4612a\" (UID: \"8d799407-1164-4be0-943e-6d1f73e4612a\") "
Jul 06 20:53:13.203766 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.203738    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d799407-1164-4be0-943e-6d1f73e4612a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8d799407-1164-4be0-943e-6d1f73e4612a" (UID: "8d799407-1164-4be0-943e-6d1f73e4612a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:13.205598 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.205571    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d799407-1164-4be0-943e-6d1f73e4612a-kube-api-access-4m7h2" (OuterVolumeSpecName: "kube-api-access-4m7h2") pod "8d799407-1164-4be0-943e-6d1f73e4612a" (UID: "8d799407-1164-4be0-943e-6d1f73e4612a"). InnerVolumeSpecName "kube-api-access-4m7h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:53:13.205740 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.205698    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d799407-1164-4be0-943e-6d1f73e4612a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d799407-1164-4be0-943e-6d1f73e4612a" (UID: "8d799407-1164-4be0-943e-6d1f73e4612a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:53:13.210183 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.210160    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d799407-1164-4be0-943e-6d1f73e4612a" containerID="11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344" exitCode=0
Jul 06 20:53:13.210306 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.210208    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"
Jul 06 20:53:13.210306 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.210246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" event={"ID":"8d799407-1164-4be0-943e-6d1f73e4612a","Type":"ContainerDied","Data":"11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344"}
Jul 06 20:53:13.210306 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.210299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc" event={"ID":"8d799407-1164-4be0-943e-6d1f73e4612a","Type":"ContainerDied","Data":"d86c6bb94f9846a168b03c3ece1b993f6d85a2ab05fedfe29febd4ea933caadf"}
Jul 06 20:53:13.210486 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.210326    2582 scope.go:117] "RemoveContainer" containerID="11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344"
Jul 06 20:53:13.221096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.221076    2582 scope.go:117] "RemoveContainer" containerID="11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344"
Jul 06 20:53:13.221432 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:53:13.221405    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344\": container with ID starting with 11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344 not found: ID does not exist" containerID="11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344"
Jul 06 20:53:13.221525 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.221440    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344"} err="failed to get container status \"11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344\": rpc error: code = NotFound desc = could not find container \"11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344\": container with ID starting with 11cb35404955615c5a05b7919e26bfa0cd4645c0cb58b54ca59dc7a1d80bf344 not found: ID does not exist"
Jul 06 20:53:13.237346 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.237314    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"]
Jul 06 20:53:13.242231 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.242207    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc"]
Jul 06 20:53:13.304891 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.304852    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d799407-1164-4be0-943e-6d1f73e4612a-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:13.304891 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.304883    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m7h2\" (UniqueName: \"kubernetes.io/projected/8d799407-1164-4be0-943e-6d1f73e4612a-kube-api-access-4m7h2\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:13.305134 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.304909    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8d799407-1164-4be0-943e-6d1f73e4612a-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:53:13.341532 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:13.341448    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d799407-1164-4be0-943e-6d1f73e4612a" path="/var/lib/kubelet/pods/8d799407-1164-4be0-943e-6d1f73e4612a/volumes"
Jul 06 20:53:21.115260 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:21.115220    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:53:21.872228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:21.872192    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:31.115586 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:31.115541    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:53:41.115579 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:41.115539    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:53:45.390575 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.390542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:45.673282 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.673189    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"]
Jul 06 20:53:45.673853 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.673818    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d799407-1164-4be0-943e-6d1f73e4612a" containerName="main"
Jul 06 20:53:45.673853 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.673846    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d799407-1164-4be0-943e-6d1f73e4612a" containerName="main"
Jul 06 20:53:45.674075 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.673881    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b692bc0-7add-4910-a45c-a62dd42a5757" containerName="main"
Jul 06 20:53:45.674075 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.673891    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b692bc0-7add-4910-a45c-a62dd42a5757" containerName="main"
Jul 06 20:53:45.674075 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.674017    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b692bc0-7add-4910-a45c-a62dd42a5757" containerName="main"
Jul 06 20:53:45.674075 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.674037    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d799407-1164-4be0-943e-6d1f73e4612a" containerName="main"
Jul 06 20:53:45.678811 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.678788    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.683339 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.683310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-rj5gr\""
Jul 06 20:53:45.683470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.683346    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 06 20:53:45.690162 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.689995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"]
Jul 06 20:53:45.696638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.696612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbp7t\" (UniqueName: \"kubernetes.io/projected/eb5e1d05-7438-42d8-a409-055539d25878-kube-api-access-pbp7t\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.696774 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.696662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb5e1d05-7438-42d8-a409-055539d25878-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.696774 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.696720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5e1d05-7438-42d8-a409-055539d25878-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.712290 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.712267    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:53:45.797608 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.797568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5e1d05-7438-42d8-a409-055539d25878-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.797800 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.797728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbp7t\" (UniqueName: \"kubernetes.io/projected/eb5e1d05-7438-42d8-a409-055539d25878-kube-api-access-pbp7t\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.797870 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.797796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb5e1d05-7438-42d8-a409-055539d25878-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.798252 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.798228    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb5e1d05-7438-42d8-a409-055539d25878-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.800332 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.800306    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5e1d05-7438-42d8-a409-055539d25878-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.806220 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.806195    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbp7t\" (UniqueName: \"kubernetes.io/projected/eb5e1d05-7438-42d8-a409-055539d25878-kube-api-access-pbp7t\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:45.992755 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:45.992717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:46.149336 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:46.149306    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"]
Jul 06 20:53:46.151128 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:53:46.151102    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5e1d05_7438_42d8_a409_055539d25878.slice/crio-1e36cda11d085e87be297155cc7284f32ebb41dff3844110a55226803d338563 WatchSource:0}: Error finding container 1e36cda11d085e87be297155cc7284f32ebb41dff3844110a55226803d338563: Status 404 returned error can't find the container with id 1e36cda11d085e87be297155cc7284f32ebb41dff3844110a55226803d338563
Jul 06 20:53:46.421734 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:46.421644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" event={"ID":"eb5e1d05-7438-42d8-a409-055539d25878","Type":"ContainerStarted","Data":"cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc"}
Jul 06 20:53:46.421734 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:46.421698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" event={"ID":"eb5e1d05-7438-42d8-a409-055539d25878","Type":"ContainerStarted","Data":"1e36cda11d085e87be297155cc7284f32ebb41dff3844110a55226803d338563"}
Jul 06 20:53:46.421734 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:46.421734    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:53:46.441480 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:46.441429    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" podStartSLOduration=1.441412562 podStartE2EDuration="1.441412562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:53:46.440553531 +0000 UTC m=+1123.774894141" watchObservedRunningTime="2026-07-06 20:53:46.441412562 +0000 UTC m=+1123.775753156"
Jul 06 20:53:51.116056 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:51.116015    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:53:54.650503 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:53:54.650462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:01.124708 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:01.124681    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:54:01.133012 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:01.132987    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:54:02.845989 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:02.845926    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"]
Jul 06 20:54:02.861861 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:02.861835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:02.919796 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:02.919766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:17.429235 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:17.429200    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:54:17.871348 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:17.871233    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" containerID="cri-o://f6cba471fa29284f0fe8cbf3a07207b248d7bf80f2fbf84afd1f0128cbcc05c4" gracePeriod=45
Jul 06 20:54:18.125734 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.125637    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 06 20:54:18.608746 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.608713    2582 generic.go:356] "Generic (PLEG): container finished" podID="52a56f93-9f85-4005-ba7b-195522086641" containerID="f6cba471fa29284f0fe8cbf3a07207b248d7bf80f2fbf84afd1f0128cbcc05c4" exitCode=0
Jul 06 20:54:18.609138 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.608783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" event={"ID":"52a56f93-9f85-4005-ba7b-195522086641","Type":"ContainerDied","Data":"f6cba471fa29284f0fe8cbf3a07207b248d7bf80f2fbf84afd1f0128cbcc05c4"}
Jul 06 20:54:18.640341 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.640319    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:54:18.808292 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808265    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-model-cache\") pod \"52a56f93-9f85-4005-ba7b-195522086641\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") "
Jul 06 20:54:18.808484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-kserve-provision-location\") pod \"52a56f93-9f85-4005-ba7b-195522086641\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") "
Jul 06 20:54:18.808484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-tmp-dir\") pod \"52a56f93-9f85-4005-ba7b-195522086641\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") "
Jul 06 20:54:18.808484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808441    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nn5x\" (UniqueName: \"kubernetes.io/projected/52a56f93-9f85-4005-ba7b-195522086641-kube-api-access-8nn5x\") pod \"52a56f93-9f85-4005-ba7b-195522086641\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") "
Jul 06 20:54:18.808484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a56f93-9f85-4005-ba7b-195522086641-tls-certs\") pod \"52a56f93-9f85-4005-ba7b-195522086641\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") "
Jul 06 20:54:18.808706 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-home\") pod \"52a56f93-9f85-4005-ba7b-195522086641\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") "
Jul 06 20:54:18.808706 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808531    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-dshm\") pod \"52a56f93-9f85-4005-ba7b-195522086641\" (UID: \"52a56f93-9f85-4005-ba7b-195522086641\") "
Jul 06 20:54:18.808706 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808576    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-model-cache" (OuterVolumeSpecName: "model-cache") pod "52a56f93-9f85-4005-ba7b-195522086641" (UID: "52a56f93-9f85-4005-ba7b-195522086641"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:18.808877 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.808839    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:54:18.809515 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.809486    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-home" (OuterVolumeSpecName: "home") pod "52a56f93-9f85-4005-ba7b-195522086641" (UID: "52a56f93-9f85-4005-ba7b-195522086641"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:18.811269 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.811239    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a56f93-9f85-4005-ba7b-195522086641-kube-api-access-8nn5x" (OuterVolumeSpecName: "kube-api-access-8nn5x") pod "52a56f93-9f85-4005-ba7b-195522086641" (UID: "52a56f93-9f85-4005-ba7b-195522086641"). InnerVolumeSpecName "kube-api-access-8nn5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:54:18.811371 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.811289    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-dshm" (OuterVolumeSpecName: "dshm") pod "52a56f93-9f85-4005-ba7b-195522086641" (UID: "52a56f93-9f85-4005-ba7b-195522086641"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:18.811492 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.811467    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a56f93-9f85-4005-ba7b-195522086641-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52a56f93-9f85-4005-ba7b-195522086641" (UID: "52a56f93-9f85-4005-ba7b-195522086641"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:54:18.821662 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.821640    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52a56f93-9f85-4005-ba7b-195522086641" (UID: "52a56f93-9f85-4005-ba7b-195522086641"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:18.876346 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.876306    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52a56f93-9f85-4005-ba7b-195522086641" (UID: "52a56f93-9f85-4005-ba7b-195522086641"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:54:18.910002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.909944    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nn5x\" (UniqueName: \"kubernetes.io/projected/52a56f93-9f85-4005-ba7b-195522086641-kube-api-access-8nn5x\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:54:18.910002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.909998    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a56f93-9f85-4005-ba7b-195522086641-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:54:18.910002 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.910009    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:54:18.910228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.910016    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:54:18.910228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.910024    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:54:18.910228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:18.910032    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a56f93-9f85-4005-ba7b-195522086641-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:54:19.615541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:19.615458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7" event={"ID":"52a56f93-9f85-4005-ba7b-195522086641","Type":"ContainerDied","Data":"ffbe6f9a53b8df26f2e4663c4bb532c0f5da775e696c2b3d79a1713d85ba5e6e"}
Jul 06 20:54:19.615541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:19.615495    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"
Jul 06 20:54:19.615541 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:19.615529    2582 scope.go:117] "RemoveContainer" containerID="f6cba471fa29284f0fe8cbf3a07207b248d7bf80f2fbf84afd1f0128cbcc05c4"
Jul 06 20:54:19.625102 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:19.624991    2582 scope.go:117] "RemoveContainer" containerID="39445e574ff91abcac9d4f39b2c504d6bc1981a24e159d1a50e2ec11424dc582"
Jul 06 20:54:19.637566 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:19.637544    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"]
Jul 06 20:54:19.641083 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:19.641052    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7"]
Jul 06 20:54:21.337890 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:21.337847    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a56f93-9f85-4005-ba7b-195522086641" path="/var/lib/kubelet/pods/52a56f93-9f85-4005-ba7b-195522086641/volumes"
Jul 06 20:54:21.805552 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:21.805522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:23.862213 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.862180    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"]
Jul 06 20:54:23.862609 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.862590    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="storage-initializer"
Jul 06 20:54:23.862609 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.862602    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="storage-initializer"
Jul 06 20:54:23.862682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.862615    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main"
Jul 06 20:54:23.862682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.862620    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main"
Jul 06 20:54:23.862749 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.862685    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a56f93-9f85-4005-ba7b-195522086641" containerName="main"
Jul 06 20:54:23.867484 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.867463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:23.870402 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.870384    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 06 20:54:23.877415 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.877393    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"]
Jul 06 20:54:23.956901 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.956870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:23.957122 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.956914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf48v\" (UniqueName: \"kubernetes.io/projected/41175753-4b5b-4ad8-a04c-865d97bed9ce-kube-api-access-rf48v\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:23.957122 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.957003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:23.957122 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.957100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:23.957250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.957198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41175753-4b5b-4ad8-a04c-865d97bed9ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:23.957250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.957231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:23.958888 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:23.958868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:24.057825 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.057785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41175753-4b5b-4ad8-a04c-865d97bed9ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.057865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.057941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058054 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.058004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf48v\" (UniqueName: \"kubernetes.io/projected/41175753-4b5b-4ad8-a04c-865d97bed9ce-kube-api-access-rf48v\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058246 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.058059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058246 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.058114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058482 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.058455    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058674 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.058648    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.058752 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.058721    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.060861 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.060817    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.061386 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.061355    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41175753-4b5b-4ad8-a04c-865d97bed9ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.067363 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.067338    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf48v\" (UniqueName: \"kubernetes.io/projected/41175753-4b5b-4ad8-a04c-865d97bed9ce-kube-api-access-rf48v\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.160525 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.160424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:24.178473 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.178440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:24.321435 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.321409    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"]
Jul 06 20:54:24.322818 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:54:24.322792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41175753_4b5b_4ad8_a04c_865d97bed9ce.slice/crio-a20a03f906f4b79d03825d9e98ce8675fdffe1133da2d39d3b41f3cb821c2e4c WatchSource:0}: Error finding container a20a03f906f4b79d03825d9e98ce8675fdffe1133da2d39d3b41f3cb821c2e4c: Status 404 returned error can't find the container with id a20a03f906f4b79d03825d9e98ce8675fdffe1133da2d39d3b41f3cb821c2e4c
Jul 06 20:54:24.651331 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.651244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd" event={"ID":"41175753-4b5b-4ad8-a04c-865d97bed9ce","Type":"ContainerStarted","Data":"4702c0baf176d6cf59d55fe10016b4c3643e474a4f8b22a8b0ebed58559dae8b"}
Jul 06 20:54:24.651331 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.651330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd" event={"ID":"41175753-4b5b-4ad8-a04c-865d97bed9ce","Type":"ContainerStarted","Data":"a20a03f906f4b79d03825d9e98ce8675fdffe1133da2d39d3b41f3cb821c2e4c"}
Jul 06 20:54:24.672873 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:24.672823    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd" podStartSLOduration=1.6728072219999999 podStartE2EDuration="1.672807222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:54:24.669470381 +0000 UTC m=+1162.003810976" watchObservedRunningTime="2026-07-06 20:54:24.672807222 +0000 UTC m=+1162.007147816"
Jul 06 20:54:25.178572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:25.178541    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:34.179490 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:34.179452    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:34.192033 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:34.192009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:34.721577 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:34.721548    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:54:49.990913 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:49.990880    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"]
Jul 06 20:54:49.994972 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:49.994937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:49.997985 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:49.997947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 06 20:54:49.998118 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:49.997955    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:54:50.007729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.007703    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"]
Jul 06 20:54:50.021447 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.021424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:50.108395 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.108359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kserve-provision-location\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.108572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.108460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfh5\" (UniqueName: \"kubernetes.io/projected/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kube-api-access-zpfh5\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.108572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.108492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tls-certs\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.108572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.108515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tmp-dir\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.108572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.108544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-model-cache\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.108572 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.108567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-dshm\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.108788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.108624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-home\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.209876 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.209832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfh5\" (UniqueName: \"kubernetes.io/projected/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kube-api-access-zpfh5\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.210072 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.209894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tls-certs\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.210072 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.209924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tmp-dir\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.210072 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.210003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-model-cache\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.210072 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.210037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-dshm\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.210288 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.210076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-home\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.210288 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.210119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kserve-provision-location\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.210605 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.210577    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-model-cache\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.211338 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.211312    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kserve-provision-location\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.211569 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.211540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tmp-dir\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.211682 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.211623    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-home\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.213233 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.212750    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-dshm\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.213233 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.213171    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tls-certs\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.221497 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.221472    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfh5\" (UniqueName: \"kubernetes.io/projected/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kube-api-access-zpfh5\") pod \"auth-disabled-test-kserve-6cf96f5c75-xnsm8\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.248435 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.248371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:54:50.308012 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.307948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:54:50.457285 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.457253    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"]
Jul 06 20:54:50.458623 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:54:50.458596    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb402736f_4e7e_4308_aa63_bc2c10fb37e0.slice/crio-4626dd09d5f59bd70f73428ec9ebac80da94ac662462fb6fd2ff6b9022e7f8e2 WatchSource:0}: Error finding container 4626dd09d5f59bd70f73428ec9ebac80da94ac662462fb6fd2ff6b9022e7f8e2: Status 404 returned error can't find the container with id 4626dd09d5f59bd70f73428ec9ebac80da94ac662462fb6fd2ff6b9022e7f8e2
Jul 06 20:54:50.808713 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.808620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" event={"ID":"b402736f-4e7e-4308-aa63-bc2c10fb37e0","Type":"ContainerStarted","Data":"d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1"}
Jul 06 20:54:50.808713 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:50.808681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" event={"ID":"b402736f-4e7e-4308-aa63-bc2c10fb37e0","Type":"ContainerStarted","Data":"4626dd09d5f59bd70f73428ec9ebac80da94ac662462fb6fd2ff6b9022e7f8e2"}
Jul 06 20:54:54.836784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:54.836694    2582 generic.go:356] "Generic (PLEG): container finished" podID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerID="d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1" exitCode=0
Jul 06 20:54:54.836784 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:54.836758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" event={"ID":"b402736f-4e7e-4308-aa63-bc2c10fb37e0","Type":"ContainerDied","Data":"d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1"}
Jul 06 20:54:55.844805 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:55.844774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" event={"ID":"b402736f-4e7e-4308-aa63-bc2c10fb37e0","Type":"ContainerStarted","Data":"72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa"}
Jul 06 20:54:55.871671 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:55.871615    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podStartSLOduration=2.830719653 podStartE2EDuration="6.871598496s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.040878843s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:54:55.868443195 +0000 UTC m=+1193.202783820" watchObservedRunningTime="2026-07-06 20:54:55.871598496 +0000 UTC m=+1193.205939091"
Jul 06 20:54:56.308285 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:54:56.308249    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:55:00.308646 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:00.308612    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:55:00.310347 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:00.310317    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:55:03.211327 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:03.211297    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:55:03.218676 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:03.218653    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 20:55:10.308598 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:10.308551    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:55:20.308717 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:20.308674    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:55:27.308923 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:27.308885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:30.309391 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:30.309338    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:55:40.309262 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:40.309214    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:55:42.631413 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:42.631374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:45.080028 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:45.079991    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"]
Jul 06 20:55:45.097821 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:45.097788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:45.163654 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:45.163623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:48.925315 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:48.925282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:49.026420 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:49.026393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:55:50.308467 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:55:50.308427    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:56:00.103378 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.103283    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd" podUID="41175753-4b5b-4ad8-a04c-865d97bed9ce" containerName="main" containerID="cri-o://4702c0baf176d6cf59d55fe10016b4c3643e474a4f8b22a8b0ebed58559dae8b" gracePeriod=45
Jul 06 20:56:00.237837 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.237794    2582 generic.go:356] "Generic (PLEG): container finished" podID="41175753-4b5b-4ad8-a04c-865d97bed9ce" containerID="4702c0baf176d6cf59d55fe10016b4c3643e474a4f8b22a8b0ebed58559dae8b" exitCode=0
Jul 06 20:56:00.238012 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.237863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd" event={"ID":"41175753-4b5b-4ad8-a04c-865d97bed9ce","Type":"ContainerDied","Data":"4702c0baf176d6cf59d55fe10016b4c3643e474a4f8b22a8b0ebed58559dae8b"}
Jul 06 20:56:00.308842 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.308779    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:56:00.358070 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.358010    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:56:00.502175 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502141    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-dshm\") pod \"41175753-4b5b-4ad8-a04c-865d97bed9ce\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") "
Jul 06 20:56:00.502380 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502196    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-model-cache\") pod \"41175753-4b5b-4ad8-a04c-865d97bed9ce\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") "
Jul 06 20:56:00.502380 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-tmp-dir\") pod \"41175753-4b5b-4ad8-a04c-865d97bed9ce\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") "
Jul 06 20:56:00.502380 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf48v\" (UniqueName: \"kubernetes.io/projected/41175753-4b5b-4ad8-a04c-865d97bed9ce-kube-api-access-rf48v\") pod \"41175753-4b5b-4ad8-a04c-865d97bed9ce\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") "
Jul 06 20:56:00.502557 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41175753-4b5b-4ad8-a04c-865d97bed9ce-tls-certs\") pod \"41175753-4b5b-4ad8-a04c-865d97bed9ce\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") "
Jul 06 20:56:00.502557 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502444    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-home\") pod \"41175753-4b5b-4ad8-a04c-865d97bed9ce\" (UID: \"41175753-4b5b-4ad8-a04c-865d97bed9ce\") "
Jul 06 20:56:00.502557 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502448    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "41175753-4b5b-4ad8-a04c-865d97bed9ce" (UID: "41175753-4b5b-4ad8-a04c-865d97bed9ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:00.502557 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502499    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41175753-4b5b-4ad8-a04c-865d97bed9ce" (UID: "41175753-4b5b-4ad8-a04c-865d97bed9ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:00.502767 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502685    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-home" (OuterVolumeSpecName: "home") pod "41175753-4b5b-4ad8-a04c-865d97bed9ce" (UID: "41175753-4b5b-4ad8-a04c-865d97bed9ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:00.502767 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502753    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:00.502846 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.502772    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:00.504384 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.504361    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41175753-4b5b-4ad8-a04c-865d97bed9ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41175753-4b5b-4ad8-a04c-865d97bed9ce" (UID: "41175753-4b5b-4ad8-a04c-865d97bed9ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:56:00.504741 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.504728    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-dshm" (OuterVolumeSpecName: "dshm") pod "41175753-4b5b-4ad8-a04c-865d97bed9ce" (UID: "41175753-4b5b-4ad8-a04c-865d97bed9ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:00.504804 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.504783    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41175753-4b5b-4ad8-a04c-865d97bed9ce-kube-api-access-rf48v" (OuterVolumeSpecName: "kube-api-access-rf48v") pod "41175753-4b5b-4ad8-a04c-865d97bed9ce" (UID: "41175753-4b5b-4ad8-a04c-865d97bed9ce"). InnerVolumeSpecName "kube-api-access-rf48v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:56:00.604281 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.604245    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf48v\" (UniqueName: \"kubernetes.io/projected/41175753-4b5b-4ad8-a04c-865d97bed9ce-kube-api-access-rf48v\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:00.604281 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.604275    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41175753-4b5b-4ad8-a04c-865d97bed9ce-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:00.604281 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.604286    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:00.604510 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:00.604294    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41175753-4b5b-4ad8-a04c-865d97bed9ce-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:01.246470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:01.246440    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"
Jul 06 20:56:01.246470 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:01.246455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd" event={"ID":"41175753-4b5b-4ad8-a04c-865d97bed9ce","Type":"ContainerDied","Data":"a20a03f906f4b79d03825d9e98ce8675fdffe1133da2d39d3b41f3cb821c2e4c"}
Jul 06 20:56:01.246986 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:01.246519    2582 scope.go:117] "RemoveContainer" containerID="4702c0baf176d6cf59d55fe10016b4c3643e474a4f8b22a8b0ebed58559dae8b"
Jul 06 20:56:01.271633 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:01.271599    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"]
Jul 06 20:56:01.275812 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:01.275788    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd"]
Jul 06 20:56:01.337813 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:01.337777    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41175753-4b5b-4ad8-a04c-865d97bed9ce" path="/var/lib/kubelet/pods/41175753-4b5b-4ad8-a04c-865d97bed9ce/volumes"
Jul 06 20:56:06.103729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.103692    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"]
Jul 06 20:56:06.113313 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:56:06.113272    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 20:56:06.113313 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:56:06.113313 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:56:06.113313 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc" cmd=["/bin/sleep","15"]
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.113332    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" message="" err=<
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:56:06.113349    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" podUID="eb5e1d05-7438-42d8-a409-055539d25878" containerName="main" containerID="cri-o://cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc"
Jul 06 20:56:06.113638 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.113407    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" podUID="eb5e1d05-7438-42d8-a409-055539d25878" containerName="main" containerID="cri-o://cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc" gracePeriod=60
Jul 06 20:56:06.128519 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.128439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:06.170379 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:56:06.170341    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5e1d05_7438_42d8_a409_055539d25878.slice/crio-conmon-cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc.scope\": RecentStats: unable to find data in memory cache]"
Jul 06 20:56:06.195100 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.195033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:06.284839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.284799    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb5e1d05-7438-42d8-a409-055539d25878" containerID="cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc" exitCode=0
Jul 06 20:56:06.285070 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.284890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" event={"ID":"eb5e1d05-7438-42d8-a409-055539d25878","Type":"ContainerDied","Data":"cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc"}
Jul 06 20:56:06.397312 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.397273    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:56:06.561550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.561505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb5e1d05-7438-42d8-a409-055539d25878-tokenizer-uds\") pod \"eb5e1d05-7438-42d8-a409-055539d25878\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") "
Jul 06 20:56:06.561729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.561610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5e1d05-7438-42d8-a409-055539d25878-tls-certs\") pod \"eb5e1d05-7438-42d8-a409-055539d25878\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") "
Jul 06 20:56:06.561729 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.561652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbp7t\" (UniqueName: \"kubernetes.io/projected/eb5e1d05-7438-42d8-a409-055539d25878-kube-api-access-pbp7t\") pod \"eb5e1d05-7438-42d8-a409-055539d25878\" (UID: \"eb5e1d05-7438-42d8-a409-055539d25878\") "
Jul 06 20:56:06.561861 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.561750    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5e1d05-7438-42d8-a409-055539d25878-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "eb5e1d05-7438-42d8-a409-055539d25878" (UID: "eb5e1d05-7438-42d8-a409-055539d25878"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:06.561928 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.561911    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eb5e1d05-7438-42d8-a409-055539d25878-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:06.563766 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.563731    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5e1d05-7438-42d8-a409-055539d25878-kube-api-access-pbp7t" (OuterVolumeSpecName: "kube-api-access-pbp7t") pod "eb5e1d05-7438-42d8-a409-055539d25878" (UID: "eb5e1d05-7438-42d8-a409-055539d25878"). InnerVolumeSpecName "kube-api-access-pbp7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:56:06.563889 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.563834    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb5e1d05-7438-42d8-a409-055539d25878-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb5e1d05-7438-42d8-a409-055539d25878" (UID: "eb5e1d05-7438-42d8-a409-055539d25878"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:56:06.663296 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.663201    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5e1d05-7438-42d8-a409-055539d25878-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:06.663296 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:06.663251    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbp7t\" (UniqueName: \"kubernetes.io/projected/eb5e1d05-7438-42d8-a409-055539d25878-kube-api-access-pbp7t\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:56:07.291392 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:07.291367    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"
Jul 06 20:56:07.291843 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:07.291409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln" event={"ID":"eb5e1d05-7438-42d8-a409-055539d25878","Type":"ContainerDied","Data":"1e36cda11d085e87be297155cc7284f32ebb41dff3844110a55226803d338563"}
Jul 06 20:56:07.291843 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:07.291465    2582 scope.go:117] "RemoveContainer" containerID="cf63f210bdd0a6708a8a7755f06fba37168bbd0a8749e3c551d0f3c8bcbe29bc"
Jul 06 20:56:07.321389 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:07.321336    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"]
Jul 06 20:56:07.326026 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:07.325952    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln"]
Jul 06 20:56:07.339280 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:07.339247    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb5e1d05-7438-42d8-a409-055539d25878" path="/var/lib/kubelet/pods/eb5e1d05-7438-42d8-a409-055539d25878/volumes"
Jul 06 20:56:10.309788 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:10.309741    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:56:20.308671 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:20.308626    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:56:30.308737 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:30.308691    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:56:40.308766 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:40.308721    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 06 20:56:50.319362 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.319327    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:56:50.327231 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.327202    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:56:50.464854 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.464811    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"]
Jul 06 20:56:50.465315 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.465297    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41175753-4b5b-4ad8-a04c-865d97bed9ce" containerName="main"
Jul 06 20:56:50.465411 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.465318    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="41175753-4b5b-4ad8-a04c-865d97bed9ce" containerName="main"
Jul 06 20:56:50.465411 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.465334    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb5e1d05-7438-42d8-a409-055539d25878" containerName="main"
Jul 06 20:56:50.465411 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.465341    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb5e1d05-7438-42d8-a409-055539d25878" containerName="main"
Jul 06 20:56:50.465570 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.465448    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="41175753-4b5b-4ad8-a04c-865d97bed9ce" containerName="main"
Jul 06 20:56:50.465570 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.465461    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb5e1d05-7438-42d8-a409-055539d25878" containerName="main"
Jul 06 20:56:50.470331 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.470310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.473448 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.473421    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 06 20:56:50.473714 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.473697    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:56:50.484428 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.484401    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"]
Jul 06 20:56:50.490641 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.490615    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:50.593994 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.593894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.593994 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.593931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5m2r\" (UniqueName: \"kubernetes.io/projected/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kube-api-access-c5m2r\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.594192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.593996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.594192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.594014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.594192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.594059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.594192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.594112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.594192 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.594144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.594372 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.594205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695249 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695249 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695528 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695528 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695528 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695528 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695528 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.695777 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.695553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5m2r\" (UniqueName: \"kubernetes.io/projected/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kube-api-access-c5m2r\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.696350 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.696324    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.696523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.696369    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.696523 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.696447    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.696709 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.696633    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.697158 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.697133    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.698802 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.698771    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.699072 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.699047    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.709419 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.709385    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5m2r\" (UniqueName: \"kubernetes.io/projected/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kube-api-access-c5m2r\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.721699 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.721667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:56:50.781555 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.781520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:56:50.939993 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:50.939951    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"]
Jul 06 20:56:50.940712 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:56:50.940688    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fb2823_9435_4ddd_b1e6_7ea4d100c0aa.slice/crio-95e9568db6f8bdf2a87244940c667cdca132d07a21442ac06292ee7c7227af83 WatchSource:0}: Error finding container 95e9568db6f8bdf2a87244940c667cdca132d07a21442ac06292ee7c7227af83: Status 404 returned error can't find the container with id 95e9568db6f8bdf2a87244940c667cdca132d07a21442ac06292ee7c7227af83
Jul 06 20:56:51.546473 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:51.546438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" event={"ID":"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa","Type":"ContainerStarted","Data":"462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022"}
Jul 06 20:56:51.546888 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:51.546508    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" event={"ID":"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa","Type":"ContainerStarted","Data":"95e9568db6f8bdf2a87244940c667cdca132d07a21442ac06292ee7c7227af83"}
Jul 06 20:56:55.573914 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:55.573872    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerID="462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022" exitCode=0
Jul 06 20:56:55.574460 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:55.573946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" event={"ID":"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa","Type":"ContainerDied","Data":"462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022"}
Jul 06 20:56:56.582240 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:56.582191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" event={"ID":"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa","Type":"ContainerStarted","Data":"32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841"}
Jul 06 20:56:56.609079 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:56.608981    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podStartSLOduration=2.666479797 podStartE2EDuration="6.608940463s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.942460666s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:56:56.607174191 +0000 UTC m=+1313.941514797" watchObservedRunningTime="2026-07-06 20:56:56.608940463 +0000 UTC m=+1313.943281059"
Jul 06 20:56:56.781881 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:56:56.781850    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:57:00.782128 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:00.782090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:57:00.783644 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:00.783613    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:57:03.508592 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:03.508557    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:05.239254 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:05.239215    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:05.241791 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:05.241764    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"]
Jul 06 20:57:05.301337 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:05.301306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:57:10.782903 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:10.782845    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:57:17.716340 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:17.716306    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="2f068bd56567e2c311987dec49318f1aae94721ebc31038b46bcb24f1bcf3ea7" exitCode=137
Jul 06 20:57:17.716791 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:17.716380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"2f068bd56567e2c311987dec49318f1aae94721ebc31038b46bcb24f1bcf3ea7"}
Jul 06 20:57:17.716791 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:17.716445    2582 scope.go:117] "RemoveContainer" containerID="b7e7f760ad637870ff4c58e43d1480f88326c91579dfa567ba3d521fb9f7ddf2"
Jul 06 20:57:17.716900 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:17.716827    2582 scope.go:117] "RemoveContainer" containerID="2f068bd56567e2c311987dec49318f1aae94721ebc31038b46bcb24f1bcf3ea7"
Jul 06 20:57:17.717075 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:57:17.717051    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 20:57:20.266651 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:20.266602    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main" containerID="cri-o://72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa" gracePeriod=45
Jul 06 20:57:20.782384 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:20.782337    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:57:21.028418 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.028392    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:57:21.218922 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.218890    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-home\") pod \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") "
Jul 06 20:57:21.219131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.218946    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-dshm\") pod \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") "
Jul 06 20:57:21.219131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.218989    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kserve-provision-location\") pod \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") "
Jul 06 20:57:21.219131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.219023    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tls-certs\") pod \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") "
Jul 06 20:57:21.219131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.219053    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tmp-dir\") pod \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") "
Jul 06 20:57:21.219131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.219087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-model-cache\") pod \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") "
Jul 06 20:57:21.219131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.219123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpfh5\" (UniqueName: \"kubernetes.io/projected/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kube-api-access-zpfh5\") pod \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\" (UID: \"b402736f-4e7e-4308-aa63-bc2c10fb37e0\") "
Jul 06 20:57:21.219554 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.219523    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b402736f-4e7e-4308-aa63-bc2c10fb37e0" (UID: "b402736f-4e7e-4308-aa63-bc2c10fb37e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:57:21.219628 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.219561    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-home" (OuterVolumeSpecName: "home") pod "b402736f-4e7e-4308-aa63-bc2c10fb37e0" (UID: "b402736f-4e7e-4308-aa63-bc2c10fb37e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:57:21.221542 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.221514    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kube-api-access-zpfh5" (OuterVolumeSpecName: "kube-api-access-zpfh5") pod "b402736f-4e7e-4308-aa63-bc2c10fb37e0" (UID: "b402736f-4e7e-4308-aa63-bc2c10fb37e0"). InnerVolumeSpecName "kube-api-access-zpfh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:57:21.222121 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.222096    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-dshm" (OuterVolumeSpecName: "dshm") pod "b402736f-4e7e-4308-aa63-bc2c10fb37e0" (UID: "b402736f-4e7e-4308-aa63-bc2c10fb37e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:57:21.222214 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.222098    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b402736f-4e7e-4308-aa63-bc2c10fb37e0" (UID: "b402736f-4e7e-4308-aa63-bc2c10fb37e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:57:21.233033 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.232998    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b402736f-4e7e-4308-aa63-bc2c10fb37e0" (UID: "b402736f-4e7e-4308-aa63-bc2c10fb37e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:57:21.284102 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.284041    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b402736f-4e7e-4308-aa63-bc2c10fb37e0" (UID: "b402736f-4e7e-4308-aa63-bc2c10fb37e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:57:21.320828 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.320788    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfh5\" (UniqueName: \"kubernetes.io/projected/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kube-api-access-zpfh5\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:57:21.320828 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.320818    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:57:21.320828 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.320832    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:57:21.321096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.320844    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:57:21.321096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.320857    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:57:21.321096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.320868    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:57:21.321096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.320879    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b402736f-4e7e-4308-aa63-bc2c10fb37e0-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:57:21.744727 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.744693    2582 generic.go:356] "Generic (PLEG): container finished" podID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerID="72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa" exitCode=0
Jul 06 20:57:21.744908 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.744767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" event={"ID":"b402736f-4e7e-4308-aa63-bc2c10fb37e0","Type":"ContainerDied","Data":"72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa"}
Jul 06 20:57:21.744908 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.744784    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"
Jul 06 20:57:21.744908 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.744839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8" event={"ID":"b402736f-4e7e-4308-aa63-bc2c10fb37e0","Type":"ContainerDied","Data":"4626dd09d5f59bd70f73428ec9ebac80da94ac662462fb6fd2ff6b9022e7f8e2"}
Jul 06 20:57:21.744908 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.744863    2582 scope.go:117] "RemoveContainer" containerID="72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa"
Jul 06 20:57:21.753902 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.753884    2582 scope.go:117] "RemoveContainer" containerID="d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1"
Jul 06 20:57:21.762680 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.762665    2582 scope.go:117] "RemoveContainer" containerID="72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa"
Jul 06 20:57:21.762989 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:57:21.762944    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa\": container with ID starting with 72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa not found: ID does not exist" containerID="72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa"
Jul 06 20:57:21.763075 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.763048    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa"} err="failed to get container status \"72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa\": rpc error: code = NotFound desc = could not find container \"72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa\": container with ID starting with 72a0209cae37ecb3e5398a3e408d5be97a2b49666ba4a4c73e273072ba3bbaaa not found: ID does not exist"
Jul 06 20:57:21.763141 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.763076    2582 scope.go:117] "RemoveContainer" containerID="d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1"
Jul 06 20:57:21.763340 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:57:21.763324    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1\": container with ID starting with d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1 not found: ID does not exist" containerID="d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1"
Jul 06 20:57:21.763390 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.763342    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1"} err="failed to get container status \"d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1\": rpc error: code = NotFound desc = could not find container \"d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1\": container with ID starting with d7c6f660949f2f9655dfb20f7b586d3cf2463afe718f68cfd44cfb9f41130dc1 not found: ID does not exist"
Jul 06 20:57:21.772283 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.772258    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"]
Jul 06 20:57:21.776671 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:21.776649    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8"]
Jul 06 20:57:23.336607 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:23.336574    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" path="/var/lib/kubelet/pods/b402736f-4e7e-4308-aa63-bc2c10fb37e0/volumes"
Jul 06 20:57:25.561433 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:25.561344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:57:25.561905 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:25.561886    2582 scope.go:117] "RemoveContainer" containerID="2f068bd56567e2c311987dec49318f1aae94721ebc31038b46bcb24f1bcf3ea7"
Jul 06 20:57:25.562133 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:57:25.562110    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 20:57:30.783058 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:30.783006    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:57:39.331873 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:39.331836    2582 scope.go:117] "RemoveContainer" containerID="2f068bd56567e2c311987dec49318f1aae94721ebc31038b46bcb24f1bcf3ea7"
Jul 06 20:57:39.333326 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:39.333294    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:57:39.333459 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:39.333383    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:57:39.333459 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:39.333435    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:57:39.860012 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:39.859943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"5ccd8f2f255c9d7706d751dadb4a6c85858ab3cc154e7aa71e957fea96088b5f"}
Jul 06 20:57:40.555742 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:40.555707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:57:40.557131 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:40.557100    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:57:40.557252 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:40.557177    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:57:40.561350 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:40.561325    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:57:40.782250 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:40.782211    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:57:41.556121 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:41.556091    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:57:41.556477 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:41.556140    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:57:42.556839 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:42.556791    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:57:42.557345 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:42.556854    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:57:43.556096 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:43.556060    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:57:43.556293 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:43.556113    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:57:44.556044 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:44.556001    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:57:44.556520 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:44.556098    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:57:46.435256 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:46.435211    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:57:46.436669 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:46.436645    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:57:50.782252 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:57:50.782195    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:58:00.782592 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:00.782539    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:58:01.053102 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:01.053012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:04.035299 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:04.035262    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="5ccd8f2f255c9d7706d751dadb4a6c85858ab3cc154e7aa71e957fea96088b5f" exitCode=137
Jul 06 20:58:04.035907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:04.035342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"5ccd8f2f255c9d7706d751dadb4a6c85858ab3cc154e7aa71e957fea96088b5f"}
Jul 06 20:58:04.035907 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:04.035418    2582 scope.go:117] "RemoveContainer" containerID="2f068bd56567e2c311987dec49318f1aae94721ebc31038b46bcb24f1bcf3ea7"
Jul 06 20:58:04.036063 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:04.035997    2582 scope.go:117] "RemoveContainer" containerID="5ccd8f2f255c9d7706d751dadb4a6c85858ab3cc154e7aa71e957fea96088b5f"
Jul 06 20:58:04.036227 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:58:04.036200    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 20:58:10.783072 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:10.783025    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 06 20:58:12.089262 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:12.089228    2582 generic.go:356] "Generic (PLEG): container finished" podID="ee146378-761d-47dd-96b5-24f9cfaff899" containerID="8501700fe4571549ff53433f47ca4f02cfcf6e29d9da48b7e3b6da7b01f1b2a1" exitCode=0
Jul 06 20:58:12.089742 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:12.089297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerDied","Data":"8501700fe4571549ff53433f47ca4f02cfcf6e29d9da48b7e3b6da7b01f1b2a1"}
Jul 06 20:58:12.089742 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:12.089366    2582 scope.go:117] "RemoveContainer" containerID="60cc2f8c9c8fc1f646c263943928c0eef23e8a38494e6ae63c6121eb5c3d58ed"
Jul 06 20:58:12.089862 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:12.089816    2582 scope.go:117] "RemoveContainer" containerID="8501700fe4571549ff53433f47ca4f02cfcf6e29d9da48b7e3b6da7b01f1b2a1"
Jul 06 20:58:12.090109 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:58:12.090080    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899"
Jul 06 20:58:16.436290 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:16.436256    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:58:16.436799 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:16.436779    2582 scope.go:117] "RemoveContainer" containerID="5ccd8f2f255c9d7706d751dadb4a6c85858ab3cc154e7aa71e957fea96088b5f"
Jul 06 20:58:16.437939 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:16.437905    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:58:16.438036 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:16.438014    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:58:16.438081 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:16.438069    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:58:17.133837 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:17.133801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33"}
Jul 06 20:58:17.556181 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:17.556145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:58:17.556604 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:17.556209    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:58:17.556691 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:17.556674    2582 scope.go:117] "RemoveContainer" containerID="8501700fe4571549ff53433f47ca4f02cfcf6e29d9da48b7e3b6da7b01f1b2a1"
Jul 06 20:58:17.556899 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:58:17.556874    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899"
Jul 06 20:58:17.557485 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:17.557461    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:58:17.557588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:17.557509    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:58:18.556098 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:18.556065    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:58:18.556289 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:18.556118    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:58:19.556228 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:19.556174    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:58:19.556439 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:19.556246    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:58:20.556847 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:20.556808    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:58:20.557261 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:20.556870    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:58:20.792499 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:20.792468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:58:20.811027 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:20.810907    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:58:21.556761 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:21.556728    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:58:21.556937 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:21.556795    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:58:22.556721 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:22.556681    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 20:58:22.556909 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:22.556740    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 20:58:24.261150 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:24.261120    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:58:24.261702 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:24.261501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:58:24.262457 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:24.262437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 20:58:27.624064 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:27.623999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:30.331709 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:30.331680    2582 scope.go:117] "RemoveContainer" containerID="8501700fe4571549ff53433f47ca4f02cfcf6e29d9da48b7e3b6da7b01f1b2a1"
Jul 06 20:58:30.332122 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:58:30.331854    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899)\"" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899"
Jul 06 20:58:42.331289 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:42.331257    2582 scope.go:117] "RemoveContainer" containerID="8501700fe4571549ff53433f47ca4f02cfcf6e29d9da48b7e3b6da7b01f1b2a1"
Jul 06 20:58:42.332656 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:42.332615    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:58:42.332753 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:42.332712    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:58:42.332794 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:42.332760    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 20:58:43.290659 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:43.290612    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" event={"ID":"ee146378-761d-47dd-96b5-24f9cfaff899","Type":"ContainerStarted","Data":"53fc2d0c52ac4ca8bf418094afe197db0166a34377452f9bf85f102276f50ff2"}
Jul 06 20:58:43.769095 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:43.769060    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:58:43.770365 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:43.770328    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 06 20:58:44.769006 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:44.768943    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 06 20:58:45.769373 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:45.769324    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 06 20:58:46.769384 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:46.769330    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 06 20:58:47.555989 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:47.555935    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:58:47.769073 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:47.769008    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd" podUID="ee146378-761d-47dd-96b5-24f9cfaff899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 06 20:58:48.894550 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:48.894504    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:58:48.899518 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:48.899494    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd"
Jul 06 20:58:50.189146 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:50.189090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:58:50.191941 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:50.191911    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"]
Jul 06 20:58:50.256869 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:58:50.256834    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:05.218123 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.218033    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" containerID="cri-o://32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841" gracePeriod=45
Jul 06 20:59:05.881435 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.881346    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:59:05.967463 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967430    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-model-cache\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.967463 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967461    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-dshm\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.967669 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-cabundle-cert\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.967669 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967584    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-home\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.967669 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tmp-dir\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.967669 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967642    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tls-certs\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.967870 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kserve-provision-location\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.967870 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967736    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:59:05.967870 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.967747    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5m2r\" (UniqueName: \"kubernetes.io/projected/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kube-api-access-c5m2r\") pod \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\" (UID: \"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa\") "
Jul 06 20:59:05.968062 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.968029    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:59:05.968128 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.968067    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-home" (OuterVolumeSpecName: "home") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:59:05.968223 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.968200    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-cabundle-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:05.968311 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.968227    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:05.968311 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.968239    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:05.969906 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.969879    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:59:05.970112 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.970095    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-dshm" (OuterVolumeSpecName: "dshm") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:59:05.970408 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.970387    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kube-api-access-c5m2r" (OuterVolumeSpecName: "kube-api-access-c5m2r") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "kube-api-access-c5m2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:59:05.984675 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:05.984639    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:59:06.026851 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.026820    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" (UID: "c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:59:06.069503 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.069477    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:06.069503 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.069502    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:06.069626 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.069513    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:06.069626 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.069523    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5m2r\" (UniqueName: \"kubernetes.io/projected/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-kube-api-access-c5m2r\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:06.069626 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.069531    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 20:59:06.450646 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.450607    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerID="32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841" exitCode=0
Jul 06 20:59:06.451043 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.450666    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"
Jul 06 20:59:06.451043 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.450676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" event={"ID":"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa","Type":"ContainerDied","Data":"32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841"}
Jul 06 20:59:06.451043 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.450755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" event={"ID":"c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa","Type":"ContainerDied","Data":"95e9568db6f8bdf2a87244940c667cdca132d07a21442ac06292ee7c7227af83"}
Jul 06 20:59:06.451043 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.450777    2582 scope.go:117] "RemoveContainer" containerID="32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841"
Jul 06 20:59:06.460720 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.460702    2582 scope.go:117] "RemoveContainer" containerID="462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022"
Jul 06 20:59:06.469588 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.469576    2582 scope.go:117] "RemoveContainer" containerID="32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841"
Jul 06 20:59:06.469814 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:59:06.469794    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841\": container with ID starting with 32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841 not found: ID does not exist" containerID="32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841"
Jul 06 20:59:06.469859 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.469822    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841"} err="failed to get container status \"32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841\": rpc error: code = NotFound desc = could not find container \"32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841\": container with ID starting with 32a2656d99fc156e672ce92bb4771d537c97a1411b7d561c8dfc4685104ef841 not found: ID does not exist"
Jul 06 20:59:06.469859 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.469839    2582 scope.go:117] "RemoveContainer" containerID="462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022"
Jul 06 20:59:06.470079 ip-10-0-131-57 kubenswrapper[2582]: E0706 20:59:06.470061    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022\": container with ID starting with 462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022 not found: ID does not exist" containerID="462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022"
Jul 06 20:59:06.470144 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.470088    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022"} err="failed to get container status \"462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022\": rpc error: code = NotFound desc = could not find container \"462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022\": container with ID starting with 462f3f30bf80a4554f66155709f44abf729a980a37a632c6f2f4bfe1ca69b022 not found: ID does not exist"
Jul 06 20:59:06.475865 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.475844    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"]
Jul 06 20:59:06.480922 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.480901    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf"]
Jul 06 20:59:06.794067 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:06.793956    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Jul 06 20:59:07.336711 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:07.336670    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" path="/var/lib/kubelet/pods/c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa/volumes"
Jul 06 20:59:31.780802 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.780767    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"]
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781204    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main"
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781218    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main"
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781231    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main"
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781236    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main"
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781254    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="storage-initializer"
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781260    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="storage-initializer"
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781271    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="storage-initializer"
Jul 06 20:59:31.781319 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781276    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="storage-initializer"
Jul 06 20:59:31.781671 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781332    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5fb2823-9435-4ddd-b1e6-7ea4d100c0aa" containerName="main"
Jul 06 20:59:31.781671 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.781342    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b402736f-4e7e-4308-aa63-bc2c10fb37e0" containerName="main"
Jul 06 20:59:31.786625 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.786601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.789666 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.789639    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:59:31.789826 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.789682    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 20:59:31.789826 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.789682    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:59:31.789826 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.789745    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 06 20:59:31.798086 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.798056    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"]
Jul 06 20:59:31.800058 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.800029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:31.907727 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.907681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e14b90a5-7f03-4810-8e52-1259565af089-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.908044 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.908022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14b90a5-7f03-4810-8e52-1259565af089-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.908217 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.908200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.908341 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.908325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.908502 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.908486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.908604 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.908590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.908724 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.908710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:31.908829 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:31.908815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wq9\" (UniqueName: \"kubernetes.io/projected/e14b90a5-7f03-4810-8e52-1259565af089-kube-api-access-x4wq9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.009693 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.009648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.009874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.009701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.009874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.009748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.009874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.009768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.009874 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.009795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.009917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wq9\" (UniqueName: \"kubernetes.io/projected/e14b90a5-7f03-4810-8e52-1259565af089-kube-api-access-x4wq9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.009995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e14b90a5-7f03-4810-8e52-1259565af089-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010109 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.010023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14b90a5-7f03-4810-8e52-1259565af089-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010270 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.010148    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010270 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.010192    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010270 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.010217    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010465 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.010420    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.010910 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.010886    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e14b90a5-7f03-4810-8e52-1259565af089-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.012193 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.012170    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.012385 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.012368    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14b90a5-7f03-4810-8e52-1259565af089-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.021700 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.021679    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wq9\" (UniqueName: \"kubernetes.io/projected/e14b90a5-7f03-4810-8e52-1259565af089-kube-api-access-x4wq9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.098687 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.098601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:32.208173 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.208146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:32.251795 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.251768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"]
Jul 06 20:59:32.253489 ip-10-0-131-57 kubenswrapper[2582]: W0706 20:59:32.253458    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14b90a5_7f03_4810_8e52_1259565af089.slice/crio-911b18c7889c054ac826f261e29feb610717b52b6a5297a8ac9a8d07551b9822 WatchSource:0}: Error finding container 911b18c7889c054ac826f261e29feb610717b52b6a5297a8ac9a8d07551b9822: Status 404 returned error can't find the container with id 911b18c7889c054ac826f261e29feb610717b52b6a5297a8ac9a8d07551b9822
Jul 06 20:59:32.595774 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.595718    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" event={"ID":"e14b90a5-7f03-4810-8e52-1259565af089","Type":"ContainerStarted","Data":"df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa"}
Jul 06 20:59:32.595994 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:32.595791    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" event={"ID":"e14b90a5-7f03-4810-8e52-1259565af089","Type":"ContainerStarted","Data":"911b18c7889c054ac826f261e29feb610717b52b6a5297a8ac9a8d07551b9822"}
Jul 06 20:59:36.620830 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:36.620789    2582 generic.go:356] "Generic (PLEG): container finished" podID="e14b90a5-7f03-4810-8e52-1259565af089" containerID="df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa" exitCode=0
Jul 06 20:59:36.621314 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:36.620865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" event={"ID":"e14b90a5-7f03-4810-8e52-1259565af089","Type":"ContainerDied","Data":"df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa"}
Jul 06 20:59:37.631706 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:37.631668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" event={"ID":"e14b90a5-7f03-4810-8e52-1259565af089","Type":"ContainerStarted","Data":"79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a"}
Jul 06 20:59:37.662505 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:37.662448    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podStartSLOduration=2.510024636 podStartE2EDuration="6.662427658s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.152403022s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:37.658153164 +0000 UTC m=+1474.992493760" watchObservedRunningTime="2026-07-06 20:59:37.662427658 +0000 UTC m=+1474.996768271"
Jul 06 20:59:38.099387 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:38.099341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:40.374917 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:40.374876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 20:59:42.098806 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:42.098766    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 20:59:42.100673 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:42.100629    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 20:59:52.100189 ip-10-0-131-57 kubenswrapper[2582]: I0706 20:59:52.100137    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:00:02.100113 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:02.100067    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:00:03.271985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:03.270706    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:00:03.285456 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:03.285432    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:00:09.263194 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:09.263158    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:00:09.263624 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:09.263220    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:00:09.839033 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:09.838996    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33" exitCode=137
Jul 06 21:00:09.839201 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:09.839069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33"}
Jul 06 21:00:09.839201 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:09.839128    2582 scope.go:117] "RemoveContainer" containerID="5ccd8f2f255c9d7706d751dadb4a6c85858ab3cc154e7aa71e957fea96088b5f"
Jul 06 21:00:09.839579 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:09.839561    2582 scope.go:117] "RemoveContainer" containerID="4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33"
Jul 06 21:00:09.839777 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:00:09.839758    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:00:12.099040 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:12.098996    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:00:15.208428 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:15.208390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:21.331511 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:21.331476    2582 scope.go:117] "RemoveContainer" containerID="4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33"
Jul 06 21:00:21.331946 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:00:21.331656    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:00:22.099450 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:22.099393    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:00:24.261608 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:24.261569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:00:24.262051 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:24.262036    2582 scope.go:117] "RemoveContainer" containerID="4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33"
Jul 06 21:00:24.262262 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:00:24.262243    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:00:30.911210 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:30.911168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:00:32.099648 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:32.099609    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:00:36.331444 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:36.331402    2582 scope.go:117] "RemoveContainer" containerID="4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33"
Jul 06 21:00:36.332623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:36.332584    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:00:36.332733 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:36.332683    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:00:36.332797 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:36.332734    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:00:37.009432 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:37.009385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"60e96a4c0c77e7a20a8252f135790aaadaf49e8b997d1902594ce4aa12b7894f"}
Jul 06 21:00:37.555830 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:37.555792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:00:37.557016 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:37.556986    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:00:37.557144 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:37.557048    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:00:38.555797 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:38.555761    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:00:38.556040 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:38.555817    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:00:39.262219 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:39.262185    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:00:39.556636 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:39.556526    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:00:39.556636 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:39.556592    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:00:40.556831 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:40.556786    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:00:40.557393 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:40.556846    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:00:41.556260 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:41.556229    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:00:41.556453 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:41.556289    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:00:42.100003 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:42.099923    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:00:43.342187 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:43.342149    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:00:43.343540 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:43.343519    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:00:52.099413 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:00:52.099364    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:01:02.109727 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:02.109696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 21:01:02.118010 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:02.117979    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 21:01:03.852563 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:03.852519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:03.861990 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:03.856614    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"]
Jul 06 21:01:03.949472 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:03.949432    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:18.885039 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:18.884991    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" containerID="cri-o://79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a" gracePeriod=45
Jul 06 21:01:19.110768 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.110724    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 06 21:01:19.447405 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.447382    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 21:01:19.531558 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531466    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4wq9\" (UniqueName: \"kubernetes.io/projected/e14b90a5-7f03-4810-8e52-1259565af089-kube-api-access-x4wq9\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.531558 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531509    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-kserve-provision-location\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.531558 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-tmp-dir\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.531558 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531550    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-home\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.531914 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-dshm\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.531914 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531615    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e14b90a5-7f03-4810-8e52-1259565af089-cabundle-cert\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.531914 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531673    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-model-cache\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.531914 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531750    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14b90a5-7f03-4810-8e52-1259565af089-tls-certs\") pod \"e14b90a5-7f03-4810-8e52-1259565af089\" (UID: \"e14b90a5-7f03-4810-8e52-1259565af089\") "
Jul 06 21:01:19.532135 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.531933    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:19.532135 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.532015    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-home" (OuterVolumeSpecName: "home") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:19.532135 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.532052    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14b90a5-7f03-4810-8e52-1259565af089-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:01:19.532135 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.532079    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:19.533241 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.533207    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-model-cache" (OuterVolumeSpecName: "model-cache") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:19.533789 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.533762    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-dshm" (OuterVolumeSpecName: "dshm") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:19.534115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.534095    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14b90a5-7f03-4810-8e52-1259565af089-kube-api-access-x4wq9" (OuterVolumeSpecName: "kube-api-access-x4wq9") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "kube-api-access-x4wq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:01:19.534737 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.534719    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14b90a5-7f03-4810-8e52-1259565af089-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:01:19.599235 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.599196    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e14b90a5-7f03-4810-8e52-1259565af089" (UID: "e14b90a5-7f03-4810-8e52-1259565af089"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:19.633013 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.632984    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wq9\" (UniqueName: \"kubernetes.io/projected/e14b90a5-7f03-4810-8e52-1259565af089-kube-api-access-x4wq9\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:19.633103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.633025    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:19.633103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.633036    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:19.633103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.633045    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:19.633103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.633053    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e14b90a5-7f03-4810-8e52-1259565af089-cabundle-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:19.633103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.633062    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14b90a5-7f03-4810-8e52-1259565af089-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:19.633103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:19.633069    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14b90a5-7f03-4810-8e52-1259565af089-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:20.268527 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.268490    2582 generic.go:356] "Generic (PLEG): container finished" podID="e14b90a5-7f03-4810-8e52-1259565af089" containerID="79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a" exitCode=0
Jul 06 21:01:20.268934 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.268539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" event={"ID":"e14b90a5-7f03-4810-8e52-1259565af089","Type":"ContainerDied","Data":"79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a"}
Jul 06 21:01:20.268934 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.268547    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"
Jul 06 21:01:20.268934 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.268594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j" event={"ID":"e14b90a5-7f03-4810-8e52-1259565af089","Type":"ContainerDied","Data":"911b18c7889c054ac826f261e29feb610717b52b6a5297a8ac9a8d07551b9822"}
Jul 06 21:01:20.268934 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.268624    2582 scope.go:117] "RemoveContainer" containerID="79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a"
Jul 06 21:01:20.277951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.277932    2582 scope.go:117] "RemoveContainer" containerID="df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa"
Jul 06 21:01:20.286290 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.286271    2582 scope.go:117] "RemoveContainer" containerID="79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a"
Jul 06 21:01:20.286543 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:01:20.286522    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a\": container with ID starting with 79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a not found: ID does not exist" containerID="79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a"
Jul 06 21:01:20.286623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.286555    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a"} err="failed to get container status \"79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a\": rpc error: code = NotFound desc = could not find container \"79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a\": container with ID starting with 79359f46f3eb4fbc07196764595100eab82c468aae52dfc6f755ac68ec30be1a not found: ID does not exist"
Jul 06 21:01:20.286623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.286577    2582 scope.go:117] "RemoveContainer" containerID="df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa"
Jul 06 21:01:20.286859 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:01:20.286840    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa\": container with ID starting with df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa not found: ID does not exist" containerID="df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa"
Jul 06 21:01:20.286907 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.286866    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa"} err="failed to get container status \"df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa\": rpc error: code = NotFound desc = could not find container \"df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa\": container with ID starting with df8f98e2faf654776877b3e4c1a9e3fe25b0a2ce4610b6da7708fbb2618f53aa not found: ID does not exist"
Jul 06 21:01:20.293303 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.293280    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"]
Jul 06 21:01:20.300143 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:20.300119    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j"]
Jul 06 21:01:21.337073 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:21.337044    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14b90a5-7f03-4810-8e52-1259565af089" path="/var/lib/kubelet/pods/e14b90a5-7f03-4810-8e52-1259565af089/volumes"
Jul 06 21:01:24.190567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:24.190535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:24.297681 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:24.297642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:24.369598 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:24.369562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:26.693091 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:26.693058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:26.795458 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:26.795427    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:27.242499 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.242467    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-lw446"]
Jul 06 21:01:27.242863 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.242851    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="storage-initializer"
Jul 06 21:01:27.242863 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.242864    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="storage-initializer"
Jul 06 21:01:27.242944 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.242878    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main"
Jul 06 21:01:27.242944 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.242884    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main"
Jul 06 21:01:27.243041 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.242954    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14b90a5-7f03-4810-8e52-1259565af089" containerName="main"
Jul 06 21:01:27.247392 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.247371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:27.250218 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.250195    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:01:27.251881 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.251856    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-lw446"]
Jul 06 21:01:27.306286 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.306253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtlg\" (UniqueName: \"kubernetes.io/projected/dd553b25-654c-43bd-a8d5-14ebc59a4dc8-kube-api-access-5xtlg\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:27.322375 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.322346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:27.407314 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.407278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtlg\" (UniqueName: \"kubernetes.io/projected/dd553b25-654c-43bd-a8d5-14ebc59a4dc8-kube-api-access-5xtlg\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:27.417325 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:27.417307    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtlg\" (UniqueName: \"kubernetes.io/projected/dd553b25-654c-43bd-a8d5-14ebc59a4dc8-kube-api-access-5xtlg\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.201322 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.201280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.207805 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.207780    2582 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.302715 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.302682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.302886 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.302841    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.306531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.306511    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.522673 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.522638    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/95d8fd9e3a89fba13ba92fa30ad2ea3eca1fe57a3adce27e13a21899d478620a/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.537101 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.537067    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e2e-pvc-model-download-lw446\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.757843 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.757808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:37.892119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:37.892089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-lw446"]
Jul 06 21:01:37.893506 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:01:37.893476    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd553b25_654c_43bd_a8d5_14ebc59a4dc8.slice/crio-3f99988dcf580486b539c72c43f4d4de324d4977b135a0c3694059ad6150eead WatchSource:0}: Error finding container 3f99988dcf580486b539c72c43f4d4de324d4977b135a0c3694059ad6150eead: Status 404 returned error can't find the container with id 3f99988dcf580486b539c72c43f4d4de324d4977b135a0c3694059ad6150eead
Jul 06 21:01:38.089782 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:38.089699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:38.189679 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:38.189646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:01:38.378309 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:38.378220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446" event={"ID":"dd553b25-654c-43bd-a8d5-14ebc59a4dc8","Type":"ContainerStarted","Data":"0b7729ca71af2c329f90b283b7f3bb5c6304bd89c1c728b453fb2148fa81b488"}
Jul 06 21:01:38.378309 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:38.378276    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446" event={"ID":"dd553b25-654c-43bd-a8d5-14ebc59a4dc8","Type":"ContainerStarted","Data":"3f99988dcf580486b539c72c43f4d4de324d4977b135a0c3694059ad6150eead"}
Jul 06 21:01:38.399386 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:38.399326    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446" podStartSLOduration=14.399309184 podStartE2EDuration="14.399309184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:01:38.398178302 +0000 UTC m=+1595.732518910" watchObservedRunningTime="2026-07-06 21:01:38.399309184 +0000 UTC m=+1595.733649779"
Jul 06 21:01:45.419005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:45.418938    2582 generic.go:356] "Generic (PLEG): container finished" podID="dd553b25-654c-43bd-a8d5-14ebc59a4dc8" containerID="0b7729ca71af2c329f90b283b7f3bb5c6304bd89c1c728b453fb2148fa81b488" exitCode=0
Jul 06 21:01:45.419456 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:45.419011    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446" event={"ID":"dd553b25-654c-43bd-a8d5-14ebc59a4dc8","Type":"ContainerDied","Data":"0b7729ca71af2c329f90b283b7f3bb5c6304bd89c1c728b453fb2148fa81b488"}
Jul 06 21:01:46.584057 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.584033    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:46.697754 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.697720    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xtlg\" (UniqueName: \"kubernetes.io/projected/dd553b25-654c-43bd-a8d5-14ebc59a4dc8-kube-api-access-5xtlg\") pod \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") "
Jul 06 21:01:46.697924 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.697865    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\" (UID: \"dd553b25-654c-43bd-a8d5-14ebc59a4dc8\") "
Jul 06 21:01:46.700055 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.700023    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd553b25-654c-43bd-a8d5-14ebc59a4dc8-kube-api-access-5xtlg" (OuterVolumeSpecName: "kube-api-access-5xtlg") pod "dd553b25-654c-43bd-a8d5-14ebc59a4dc8" (UID: "dd553b25-654c-43bd-a8d5-14ebc59a4dc8"). InnerVolumeSpecName "kube-api-access-5xtlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:01:46.701795 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.701774    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e" (OuterVolumeSpecName: "model-storage") pod "dd553b25-654c-43bd-a8d5-14ebc59a4dc8" (UID: "dd553b25-654c-43bd-a8d5-14ebc59a4dc8"). InnerVolumeSpecName "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:01:46.799155 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.799122    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtlg\" (UniqueName: \"kubernetes.io/projected/dd553b25-654c-43bd-a8d5-14ebc59a4dc8-kube-api-access-5xtlg\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:01:46.799301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.799173    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" "
Jul 06 21:01:46.915422 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:46.915399    2582 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e") on node "ip-10-0-131-57.ec2.internal"
Jul 06 21:01:47.001187 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:47.001104    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\""
Jul 06 21:01:47.432393 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:47.432364    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446"
Jul 06 21:01:47.432393 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:47.432391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-lw446" event={"ID":"dd553b25-654c-43bd-a8d5-14ebc59a4dc8","Type":"ContainerDied","Data":"3f99988dcf580486b539c72c43f4d4de324d4977b135a0c3694059ad6150eead"}
Jul 06 21:01:47.432605 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:01:47.432424    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f99988dcf580486b539c72c43f4d4de324d4977b135a0c3694059ad6150eead"
Jul 06 21:02:11.577581 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.577545    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"]
Jul 06 21:02:11.578203 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.578182    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd553b25-654c-43bd-a8d5-14ebc59a4dc8" containerName="storage-initializer"
Jul 06 21:02:11.578304 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.578207    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd553b25-654c-43bd-a8d5-14ebc59a4dc8" containerName="storage-initializer"
Jul 06 21:02:11.578371 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.578359    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd553b25-654c-43bd-a8d5-14ebc59a4dc8" containerName="storage-initializer"
Jul 06 21:02:11.581587 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.581571    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.584747 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.584723    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 06 21:02:11.584908 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.584894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:02:11.591237 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.591215    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"]
Jul 06 21:02:11.610176 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.610146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:11.726156 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.726116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11349a2-c3c8-419b-9246-2f0fd367362b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.726344 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.726176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.726344 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.726236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4qf\" (UniqueName: \"kubernetes.io/projected/b11349a2-c3c8-419b-9246-2f0fd367362b-kube-api-access-qr4qf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.726344 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.726259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.726344 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.726281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.726344 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.726304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827461 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827461 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11349a2-c3c8-419b-9246-2f0fd367362b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827461 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827461 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4qf\" (UniqueName: \"kubernetes.io/projected/b11349a2-c3c8-419b-9246-2f0fd367362b-kube-api-access-qr4qf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827461 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827674    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827715    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.827857 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.827782    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.829517 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.829497    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.829773 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.829754    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11349a2-c3c8-419b-9246-2f0fd367362b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.838099 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.838073    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4qf\" (UniqueName: \"kubernetes.io/projected/b11349a2-c3c8-419b-9246-2f0fd367362b-kube-api-access-qr4qf\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:11.929252 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.929216    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"]
Jul 06 21:02:11.933585 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.933562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:11.937112 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.937083    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-qxkjc\""
Jul 06 21:02:11.946539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:11.946518    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"]
Jul 06 21:02:12.029433 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.029387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0756165d-0ba9-4764-b499-8bab48d3f98b-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.029620 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.029507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbrc\" (UniqueName: \"kubernetes.io/projected/0756165d-0ba9-4764-b499-8bab48d3f98b-kube-api-access-pjbrc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.029620 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.029569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0756165d-0ba9-4764-b499-8bab48d3f98b-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.130434 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.130337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbrc\" (UniqueName: \"kubernetes.io/projected/0756165d-0ba9-4764-b499-8bab48d3f98b-kube-api-access-pjbrc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.130434 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.130402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0756165d-0ba9-4764-b499-8bab48d3f98b-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.130635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.130493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0756165d-0ba9-4764-b499-8bab48d3f98b-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.130899 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.130876    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0756165d-0ba9-4764-b499-8bab48d3f98b-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.132953 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.132928    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0756165d-0ba9-4764-b499-8bab48d3f98b-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.141227 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.141203    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbrc\" (UniqueName: \"kubernetes.io/projected/0756165d-0ba9-4764-b499-8bab48d3f98b-kube-api-access-pjbrc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.243645 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.243605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.392761 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.392730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"]
Jul 06 21:02:12.394644 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:02:12.394614    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0756165d_0ba9_4764_b499_8bab48d3f98b.slice/crio-36872212eb59775787a8718f9dc028cdebe925f297e818fa9230a95e68161736 WatchSource:0}: Error finding container 36872212eb59775787a8718f9dc028cdebe925f297e818fa9230a95e68161736: Status 404 returned error can't find the container with id 36872212eb59775787a8718f9dc028cdebe925f297e818fa9230a95e68161736
Jul 06 21:02:12.572046 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.572009    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" event={"ID":"0756165d-0ba9-4764-b499-8bab48d3f98b","Type":"ContainerStarted","Data":"3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1"}
Jul 06 21:02:12.572248 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.572069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" event={"ID":"0756165d-0ba9-4764-b499-8bab48d3f98b","Type":"ContainerStarted","Data":"36872212eb59775787a8718f9dc028cdebe925f297e818fa9230a95e68161736"}
Jul 06 21:02:12.572248 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.572198    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:12.595233 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.595173    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" podStartSLOduration=1.595158158 podStartE2EDuration="1.595158158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:02:12.592566338 +0000 UTC m=+1629.926906934" watchObservedRunningTime="2026-07-06 21:02:12.595158158 +0000 UTC m=+1629.929498776"
Jul 06 21:02:12.990205 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:12.990170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:17.885128 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:17.885093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:17.892241 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:17.892212    2582 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:17.986588 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:17.986551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:17.986775 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:17.986710    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:17.990361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:17.990344    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:18.112741 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:18.112700    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/95d8fd9e3a89fba13ba92fa30ad2ea3eca1fe57a3adce27e13a21899d478620a/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:18.137772 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:18.137698    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:18.192694 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:18.192665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:18.365039 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:18.364914    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"]
Jul 06 21:02:18.367676 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:02:18.367648    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11349a2_c3c8_419b_9246_2f0fd367362b.slice/crio-c4764abd1eba0f00137fb539e874dbbdab3447520324ef4cf35593adfadb8b65 WatchSource:0}: Error finding container c4764abd1eba0f00137fb539e874dbbdab3447520324ef4cf35593adfadb8b65: Status 404 returned error can't find the container with id c4764abd1eba0f00137fb539e874dbbdab3447520324ef4cf35593adfadb8b65
Jul 06 21:02:18.612300 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:18.612254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" event={"ID":"b11349a2-c3c8-419b-9246-2f0fd367362b","Type":"ContainerStarted","Data":"e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939"}
Jul 06 21:02:18.612496 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:18.612335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" event={"ID":"b11349a2-c3c8-419b-9246-2f0fd367362b","Type":"ContainerStarted","Data":"c4764abd1eba0f00137fb539e874dbbdab3447520324ef4cf35593adfadb8b65"}
Jul 06 21:02:18.637451 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:18.637389    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podStartSLOduration=7.637368522 podStartE2EDuration="7.637368522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:02:18.633171509 +0000 UTC m=+1635.967512116" watchObservedRunningTime="2026-07-06 21:02:18.637368522 +0000 UTC m=+1635.971709117"
Jul 06 21:02:19.192916 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:19.192884    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:28.193549 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:28.193513    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:02:28.195045 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:28.195015    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:02:38.193761 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:38.193711    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:02:43.197670 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:43.197623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:02:43.580651 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:43.580565    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:02:48.193741 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:48.193702    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:02:58.193619 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:02:58.193575    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:03:08.193697 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:03:08.193650    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:03:18.194073 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:03:18.194022    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:03:28.193909 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:03:28.193813    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:03:38.193852 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:03:38.193802    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:03:48.194224 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:03:48.194178    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:03:58.193801 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:03:58.193749    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:04:08.194207 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:08.194154    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:04:18.204121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:18.204083    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:04:18.211956 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:18.211933    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:04:22.730805 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:22.730769    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"]
Jul 06 21:04:22.733857 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:22.733825    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"]
Jul 06 21:04:22.742396 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:04:22.742343    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:04:22.742396 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:04:22.742396 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 21:04:22.742396 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1" cmd=["/bin/sleep","15"]
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:22.742758    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" message="" err=<
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:04:22.742787    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 21:04:22.742787 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" podUID="0756165d-0ba9-4764-b499-8bab48d3f98b" containerName="main" containerID="cri-o://3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1"
Jul 06 21:04:22.743241 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:22.742850    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" podUID="0756165d-0ba9-4764-b499-8bab48d3f98b" containerName="main" containerID="cri-o://3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1" gracePeriod=60
Jul 06 21:04:22.765559 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:22.765098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:22.853456 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:22.853422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:23.008617 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.008583    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:04:23.100664 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.100626    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0756165d-0ba9-4764-b499-8bab48d3f98b-tls-certs\") pod \"0756165d-0ba9-4764-b499-8bab48d3f98b\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") "
Jul 06 21:04:23.100844 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.100700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0756165d-0ba9-4764-b499-8bab48d3f98b-tokenizer-uds\") pod \"0756165d-0ba9-4764-b499-8bab48d3f98b\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") "
Jul 06 21:04:23.100844 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.100776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjbrc\" (UniqueName: \"kubernetes.io/projected/0756165d-0ba9-4764-b499-8bab48d3f98b-kube-api-access-pjbrc\") pod \"0756165d-0ba9-4764-b499-8bab48d3f98b\" (UID: \"0756165d-0ba9-4764-b499-8bab48d3f98b\") "
Jul 06 21:04:23.101004 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.100958    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0756165d-0ba9-4764-b499-8bab48d3f98b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0756165d-0ba9-4764-b499-8bab48d3f98b" (UID: "0756165d-0ba9-4764-b499-8bab48d3f98b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:04:23.101103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.101088    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0756165d-0ba9-4764-b499-8bab48d3f98b-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:23.102916 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.102893    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0756165d-0ba9-4764-b499-8bab48d3f98b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0756165d-0ba9-4764-b499-8bab48d3f98b" (UID: "0756165d-0ba9-4764-b499-8bab48d3f98b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:04:23.103001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.102910    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0756165d-0ba9-4764-b499-8bab48d3f98b-kube-api-access-pjbrc" (OuterVolumeSpecName: "kube-api-access-pjbrc") pod "0756165d-0ba9-4764-b499-8bab48d3f98b" (UID: "0756165d-0ba9-4764-b499-8bab48d3f98b"). InnerVolumeSpecName "kube-api-access-pjbrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:04:23.202538 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.202497    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0756165d-0ba9-4764-b499-8bab48d3f98b-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:23.202538 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.202535    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjbrc\" (UniqueName: \"kubernetes.io/projected/0756165d-0ba9-4764-b499-8bab48d3f98b-kube-api-access-pjbrc\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:23.365414 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.365383    2582 generic.go:356] "Generic (PLEG): container finished" podID="0756165d-0ba9-4764-b499-8bab48d3f98b" containerID="3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1" exitCode=0
Jul 06 21:04:23.365573 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.365442    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"
Jul 06 21:04:23.365573 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.365478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" event={"ID":"0756165d-0ba9-4764-b499-8bab48d3f98b","Type":"ContainerDied","Data":"3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1"}
Jul 06 21:04:23.365573 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.365527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg" event={"ID":"0756165d-0ba9-4764-b499-8bab48d3f98b","Type":"ContainerDied","Data":"36872212eb59775787a8718f9dc028cdebe925f297e818fa9230a95e68161736"}
Jul 06 21:04:23.365573 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.365541    2582 scope.go:117] "RemoveContainer" containerID="3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1"
Jul 06 21:04:23.374936 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.374919    2582 scope.go:117] "RemoveContainer" containerID="3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1"
Jul 06 21:04:23.375282 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:04:23.375260    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1\": container with ID starting with 3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1 not found: ID does not exist" containerID="3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1"
Jul 06 21:04:23.375407 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.375286    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1"} err="failed to get container status \"3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1\": rpc error: code = NotFound desc = could not find container \"3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1\": container with ID starting with 3b48683a0c98dd6163cfba990f7c20c298035bc400a5fd359138c97550704dd1 not found: ID does not exist"
Jul 06 21:04:23.385483 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.385457    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"]
Jul 06 21:04:23.389432 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:23.389409    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg"]
Jul 06 21:04:25.336776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:25.336745    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0756165d-0ba9-4764-b499-8bab48d3f98b" path="/var/lib/kubelet/pods/0756165d-0ba9-4764-b499-8bab48d3f98b/volumes"
Jul 06 21:04:37.762378 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:37.762332    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" containerID="cri-o://e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939" gracePeriod=45
Jul 06 21:04:38.205360 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.205322    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 06 21:04:38.443880 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.443856    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:04:38.445831 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.445801    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-home\") pod \"b11349a2-c3c8-419b-9246-2f0fd367362b\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") "
Jul 06 21:04:38.445997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.445876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11349a2-c3c8-419b-9246-2f0fd367362b-tls-certs\") pod \"b11349a2-c3c8-419b-9246-2f0fd367362b\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") "
Jul 06 21:04:38.445997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.445893    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-tmp-dir\") pod \"b11349a2-c3c8-419b-9246-2f0fd367362b\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") "
Jul 06 21:04:38.445997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.445936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-model-cache\") pod \"b11349a2-c3c8-419b-9246-2f0fd367362b\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") "
Jul 06 21:04:38.445997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.445984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-dshm\") pod \"b11349a2-c3c8-419b-9246-2f0fd367362b\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") "
Jul 06 21:04:38.446238 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.446008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr4qf\" (UniqueName: \"kubernetes.io/projected/b11349a2-c3c8-419b-9246-2f0fd367362b-kube-api-access-qr4qf\") pod \"b11349a2-c3c8-419b-9246-2f0fd367362b\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") "
Jul 06 21:04:38.446508 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.446469    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-home" (OuterVolumeSpecName: "home") pod "b11349a2-c3c8-419b-9246-2f0fd367362b" (UID: "b11349a2-c3c8-419b-9246-2f0fd367362b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:04:38.446634 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.446543    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b11349a2-c3c8-419b-9246-2f0fd367362b" (UID: "b11349a2-c3c8-419b-9246-2f0fd367362b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:04:38.448763 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.448726    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11349a2-c3c8-419b-9246-2f0fd367362b-kube-api-access-qr4qf" (OuterVolumeSpecName: "kube-api-access-qr4qf") pod "b11349a2-c3c8-419b-9246-2f0fd367362b" (UID: "b11349a2-c3c8-419b-9246-2f0fd367362b"). InnerVolumeSpecName "kube-api-access-qr4qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:04:38.448887 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.448731    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11349a2-c3c8-419b-9246-2f0fd367362b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b11349a2-c3c8-419b-9246-2f0fd367362b" (UID: "b11349a2-c3c8-419b-9246-2f0fd367362b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:04:38.448887 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.448838    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-dshm" (OuterVolumeSpecName: "dshm") pod "b11349a2-c3c8-419b-9246-2f0fd367362b" (UID: "b11349a2-c3c8-419b-9246-2f0fd367362b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:04:38.453738 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.453710    2582 generic.go:356] "Generic (PLEG): container finished" podID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerID="e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939" exitCode=0
Jul 06 21:04:38.453869 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.453757    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"
Jul 06 21:04:38.454009 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.453979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" event={"ID":"b11349a2-c3c8-419b-9246-2f0fd367362b","Type":"ContainerDied","Data":"e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939"}
Jul 06 21:04:38.454141 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.454049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7" event={"ID":"b11349a2-c3c8-419b-9246-2f0fd367362b","Type":"ContainerDied","Data":"c4764abd1eba0f00137fb539e874dbbdab3447520324ef4cf35593adfadb8b65"}
Jul 06 21:04:38.454141 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.454079    2582 scope.go:117] "RemoveContainer" containerID="e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939"
Jul 06 21:04:38.464658 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.464625    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b11349a2-c3c8-419b-9246-2f0fd367362b" (UID: "b11349a2-c3c8-419b-9246-2f0fd367362b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:04:38.474568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.474541    2582 scope.go:117] "RemoveContainer" containerID="e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939"
Jul 06 21:04:38.474887 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:04:38.474862    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939\": container with ID starting with e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939 not found: ID does not exist" containerID="e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939"
Jul 06 21:04:38.475019 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.474916    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939"} err="failed to get container status \"e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939\": rpc error: code = NotFound desc = could not find container \"e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939\": container with ID starting with e0a7bf5121434bd2d38a8d2150108a8da02fc4cd9c9730cb2e7be44ab37b2939 not found: ID does not exist"
Jul 06 21:04:38.546698 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.546666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"b11349a2-c3c8-419b-9246-2f0fd367362b\" (UID: \"b11349a2-c3c8-419b-9246-2f0fd367362b\") "
Jul 06 21:04:38.546913 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.546898    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:38.547013 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.546914    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:38.547013 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.546928    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr4qf\" (UniqueName: \"kubernetes.io/projected/b11349a2-c3c8-419b-9246-2f0fd367362b-kube-api-access-qr4qf\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:38.547013 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.546940    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:38.547013 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.546950    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11349a2-c3c8-419b-9246-2f0fd367362b-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:38.547013 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.546986    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11349a2-c3c8-419b-9246-2f0fd367362b-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:04:38.549328 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.549305    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e" (OuterVolumeSpecName: "kserve-pvc-source") pod "b11349a2-c3c8-419b-9246-2f0fd367362b" (UID: "b11349a2-c3c8-419b-9246-2f0fd367362b"). InnerVolumeSpecName "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:04:38.648125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.648079    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" "
Jul 06 21:04:38.676694 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.676674    2582 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e") on node "ip-10-0-131-57.ec2.internal"
Jul 06 21:04:38.748642 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.748563    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\""
Jul 06 21:04:38.782044 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.782014    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"]
Jul 06 21:04:38.785531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:38.785505    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7"]
Jul 06 21:04:39.336867 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:39.336837    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" path="/var/lib/kubelet/pods/b11349a2-c3c8-419b-9246-2f0fd367362b/volumes"
Jul 06 21:04:47.322555 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.322517    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"]
Jul 06 21:04:47.323862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.323524    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main"
Jul 06 21:04:47.323862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.323548    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main"
Jul 06 21:04:47.323862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.323649    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0756165d-0ba9-4764-b499-8bab48d3f98b" containerName="main"
Jul 06 21:04:47.323862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.323661    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0756165d-0ba9-4764-b499-8bab48d3f98b" containerName="main"
Jul 06 21:04:47.324177 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.323872    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11349a2-c3c8-419b-9246-2f0fd367362b" containerName="main"
Jul 06 21:04:47.324177 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.323895    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0756165d-0ba9-4764-b499-8bab48d3f98b" containerName="main"
Jul 06 21:04:47.329315 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.329289    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"]
Jul 06 21:04:47.329443 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.329335    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.332434 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.332396    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:04:47.332607 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.332471    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 06 21:04:47.333185 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.333162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.335839 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.335817    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-tqsnh\""
Jul 06 21:04:47.343340 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.343318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"]
Jul 06 21:04:47.343436 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.343360    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"]
Jul 06 21:04:47.353219 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.353193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:47.415551 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.415551 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.415755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.415755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.415755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.415755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.415755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.415918 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.415918 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwp7\" (UniqueName: \"kubernetes.io/projected/61301b1b-bbd6-4a84-a409-055694ddc1b5-kube-api-access-5mwp7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.415918 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f851c-fa67-48cc-b51a-b497e02561da-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.415918 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.415891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61301b1b-bbd6-4a84-a409-055694ddc1b5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.516705 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwp7\" (UniqueName: \"kubernetes.io/projected/61301b1b-bbd6-4a84-a409-055694ddc1b5-kube-api-access-5mwp7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.516909 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fvtq\" (UniqueName: \"kubernetes.io/projected/e10f851c-fa67-48cc-b51a-b497e02561da-kube-api-access-4fvtq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.516909 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f851c-fa67-48cc-b51a-b497e02561da-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.516909 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61301b1b-bbd6-4a84-a409-055694ddc1b5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.516909 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.516909 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.517301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.517301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.516951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.517301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.517301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.517301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.517301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.517623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517348    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.517623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517432    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.517623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517600    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.517795 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517640    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.517795 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517663    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.517886 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.517822    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.519236 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.519209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.519486 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.519468    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61301b1b-bbd6-4a84-a409-055694ddc1b5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.519567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.519533    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.519751 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.519734    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f851c-fa67-48cc-b51a-b497e02561da-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.525291 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.525268    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwp7\" (UniqueName: \"kubernetes.io/projected/61301b1b-bbd6-4a84-a409-055694ddc1b5-kube-api-access-5mwp7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:47.618821 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.618719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fvtq\" (UniqueName: \"kubernetes.io/projected/e10f851c-fa67-48cc-b51a-b497e02561da-kube-api-access-4fvtq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.628462 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.628428    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fvtq\" (UniqueName: \"kubernetes.io/projected/e10f851c-fa67-48cc-b51a-b497e02561da-kube-api-access-4fvtq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:47.661950 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:47.661915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:51.353743 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.353709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.359516 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.359479    2582 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.455217 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.455166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.455406 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.455306    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.458882 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.458862    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.546441 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.546408    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/95d8fd9e3a89fba13ba92fa30ad2ea3eca1fe57a3adce27e13a21899d478620a/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.564355 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.564322    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.657236 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.657150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:51.657390 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.657298    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:51.660413 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.660389    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:51.672065 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.672036    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:51.846156 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.846116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:04:51.852244 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:51.852222    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:52.009353 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:52.009252    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"]
Jul 06 21:04:52.012065 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:04:52.012034    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10f851c_fa67_48cc_b51a_b497e02561da.slice/crio-da4bdc622dac91f6ff579a043d5246bbc9e4b65e7e14b362548714a2df0a7ae8 WatchSource:0}: Error finding container da4bdc622dac91f6ff579a043d5246bbc9e4b65e7e14b362548714a2df0a7ae8: Status 404 returned error can't find the container with id da4bdc622dac91f6ff579a043d5246bbc9e4b65e7e14b362548714a2df0a7ae8
Jul 06 21:04:52.013998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:52.013981    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:04:52.026495 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:52.026473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"]
Jul 06 21:04:52.027425 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:04:52.027404    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61301b1b_bbd6_4a84_a409_055694ddc1b5.slice/crio-daa2262a2645e927fe84b4bdf5fdd40a80f53349cd308604f2c558ac219dcd50 WatchSource:0}: Error finding container daa2262a2645e927fe84b4bdf5fdd40a80f53349cd308604f2c558ac219dcd50: Status 404 returned error can't find the container with id daa2262a2645e927fe84b4bdf5fdd40a80f53349cd308604f2c558ac219dcd50
Jul 06 21:04:52.538859 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:52.538819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" event={"ID":"e10f851c-fa67-48cc-b51a-b497e02561da","Type":"ContainerStarted","Data":"da4bdc622dac91f6ff579a043d5246bbc9e4b65e7e14b362548714a2df0a7ae8"}
Jul 06 21:04:52.540623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:52.540594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" event={"ID":"61301b1b-bbd6-4a84-a409-055694ddc1b5","Type":"ContainerStarted","Data":"2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad"}
Jul 06 21:04:52.540761 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:52.540636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" event={"ID":"61301b1b-bbd6-4a84-a409-055694ddc1b5","Type":"ContainerStarted","Data":"daa2262a2645e927fe84b4bdf5fdd40a80f53349cd308604f2c558ac219dcd50"}
Jul 06 21:04:52.562128 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:52.562078    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podStartSLOduration=5.56206032 podStartE2EDuration="5.56206032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:52.560327512 +0000 UTC m=+1789.894668108" watchObservedRunningTime="2026-07-06 21:04:52.56206032 +0000 UTC m=+1789.896400949"
Jul 06 21:04:53.548863 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:53.548825    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" event={"ID":"e10f851c-fa67-48cc-b51a-b497e02561da","Type":"ContainerStarted","Data":"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"}
Jul 06 21:04:53.549260 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:53.549185    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:04:53.645015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:53.644958    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:04:54.557294 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:54.557234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" event={"ID":"e10f851c-fa67-48cc-b51a-b497e02561da","Type":"ContainerStarted","Data":"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"}
Jul 06 21:04:54.580413 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:04:54.580306    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podStartSLOduration=6.7031314779999995 podStartE2EDuration="7.580287281s" totalImagesPullingTime="877.155803ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:54.579488387 +0000 UTC m=+1791.913829006" watchObservedRunningTime="2026-07-06 21:04:54.580287281 +0000 UTC m=+1791.914627885"
Jul 06 21:05:01.847022 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:01.846985    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:05:01.847513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:01.847060    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:05:01.848951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:01.848916    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:05:01.853032 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:01.853013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:05:01.853119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:01.853074    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:05:01.854268 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:01.854243    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:05:03.321127 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:03.321093    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:05:03.329287 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:03.329255    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:05:11.847007 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:11.846930    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:05:11.853349 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:11.853311    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:05:11.865821 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:11.865795    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:05:15.866109 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:15.866069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:21.846911 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:21.846868    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:05:21.853459 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:21.853413    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:05:31.846935 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:31.846885    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:05:31.853352 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:31.853311    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:05:40.560548 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.560511    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"]
Jul 06 21:05:40.564271 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.564255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.567612 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.567582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 06 21:05:40.567752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.567644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:05:40.581559 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.581532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"]
Jul 06 21:05:40.593705 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.593674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:40.671633 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.671596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnpx\" (UniqueName: \"kubernetes.io/projected/a7ca33fd-976f-433d-81f1-9e9213134248-kube-api-access-wrnpx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.671824 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.671638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.671824 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.671667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.671824 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.671807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ca33fd-976f-433d-81f1-9e9213134248-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.671996 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.671889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.672119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.671993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.672119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.672036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773117 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnpx\" (UniqueName: \"kubernetes.io/projected/a7ca33fd-976f-433d-81f1-9e9213134248-kube-api-access-wrnpx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773117 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773378 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773378 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ca33fd-976f-433d-81f1-9e9213134248-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773378 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773550 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773742 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773720    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773839 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773721    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773839 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773826    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.773936 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.773890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.774186 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.774164    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.776018 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.775938    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.776126 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.776022    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ca33fd-976f-433d-81f1-9e9213134248-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.787645 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.787621    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnpx\" (UniqueName: \"kubernetes.io/projected/a7ca33fd-976f-433d-81f1-9e9213134248-kube-api-access-wrnpx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:40.863193 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.863111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:05:40.875174 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:40.875147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:41.053230 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:41.053197    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"]
Jul 06 21:05:41.054776 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:05:41.054748    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ca33fd_976f_433d_81f1_9e9213134248.slice/crio-09cf798507ebd479216204c4ed31d17cf0c3579e0d27696ebd16f2e1a2e507f4 WatchSource:0}: Error finding container 09cf798507ebd479216204c4ed31d17cf0c3579e0d27696ebd16f2e1a2e507f4: Status 404 returned error can't find the container with id 09cf798507ebd479216204c4ed31d17cf0c3579e0d27696ebd16f2e1a2e507f4
Jul 06 21:05:41.847214 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:41.847166    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:05:41.853670 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:41.853569    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:05:41.902269 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:41.902226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" event={"ID":"a7ca33fd-976f-433d-81f1-9e9213134248","Type":"ContainerStarted","Data":"f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885"}
Jul 06 21:05:41.902442 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:41.902302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" event={"ID":"a7ca33fd-976f-433d-81f1-9e9213134248","Type":"ContainerStarted","Data":"09cf798507ebd479216204c4ed31d17cf0c3579e0d27696ebd16f2e1a2e507f4"}
Jul 06 21:05:46.944554 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:46.944515    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7ca33fd-976f-433d-81f1-9e9213134248" containerID="f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885" exitCode=0
Jul 06 21:05:46.945084 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:46.944575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" event={"ID":"a7ca33fd-976f-433d-81f1-9e9213134248","Type":"ContainerDied","Data":"f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885"}
Jul 06 21:05:47.953767 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:47.953727    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" event={"ID":"a7ca33fd-976f-433d-81f1-9e9213134248","Type":"ContainerStarted","Data":"d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99"}
Jul 06 21:05:47.980295 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:47.980236    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podStartSLOduration=3.145370758 podStartE2EDuration="7.980216015s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.834845257s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:05:47.978870367 +0000 UTC m=+1845.313210975" watchObservedRunningTime="2026-07-06 21:05:47.980216015 +0000 UTC m=+1845.314556607"
Jul 06 21:05:48.875409 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:48.875372    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:50.876238 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:50.876197    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:05:50.878360 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:50.878315    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:05:51.846838 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:51.846794    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:05:51.853300 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:51.853245    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:05:54.865212 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:05:54.865179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:06:00.876254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:00.876194    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:06:01.847568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:01.847518    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:06:01.853001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:01.852891    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:06:10.876712 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:10.876349    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:06:11.847566 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:11.847519    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:06:11.853129 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:11.853091    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:06:20.875681 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:20.875622    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:06:21.847363 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:21.847309    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:06:21.852925 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:21.852876    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:06:30.876553 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:30.876442    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:06:31.847408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:31.847363    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:06:31.852922 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:31.852884    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:06:40.875943 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:40.875889    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:06:41.847276 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:41.847221    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:06:41.852600 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:41.852566    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:06:50.876157 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:50.876103    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:06:51.847400 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:51.847344    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:06:51.853518 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:06:51.853478    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:07:00.876510 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:00.876463    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:07:01.847191 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:01.847144    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:07:01.853138 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:01.853102    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:07:10.876164 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:10.876111    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:07:11.846822 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:11.846776    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:07:11.853192 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:11.853162    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:07:20.875804 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:20.875749    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:07:21.847297 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:21.847253    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:07:21.852825 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:21.852784    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:07:30.875812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:30.875758    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:07:31.847465 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:31.847415    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:07:31.852845 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:31.852795    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:07:40.876445 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:40.876397    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:07:41.847118 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:41.847074    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:07:41.853597 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:41.853556    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
Jul 06 21:07:50.875877 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:50.875831    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:07:51.857291 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:51.857257    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:07:51.862068 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:51.862042    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:07:51.865489 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:51.865457    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:07:51.878319 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:07:51.878294    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:08:00.876006 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:00.875881    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:08:04.999397 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:04.999363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:06.474693 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:06.474653    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"]
Jul 06 21:08:06.481752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:06.481722    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"]
Jul 06 21:08:06.519178 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:06.519150    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:06.603197 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:06.603169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:10.875628 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:10.875575    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:08:20.886157 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:20.886126    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:08:20.893884 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:20.893859    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:08:21.507002 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:21.506935    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main" containerID="cri-o://087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8" gracePeriod=45
Jul 06 21:08:21.509300 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:21.509271    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" containerID="cri-o://2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad" gracePeriod=45
Jul 06 21:08:21.858078 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:21.857972    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 06 21:08:22.323356 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.323303    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="llm-d-routing-sidecar" containerID="cri-o://db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a" gracePeriod=45
Jul 06 21:08:22.491773 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.491694    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:08:22.580860 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.580832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"61301b1b-bbd6-4a84-a409-055694ddc1b5\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") "
Jul 06 21:08:22.581064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.580874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mwp7\" (UniqueName: \"kubernetes.io/projected/61301b1b-bbd6-4a84-a409-055694ddc1b5-kube-api-access-5mwp7\") pod \"61301b1b-bbd6-4a84-a409-055694ddc1b5\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") "
Jul 06 21:08:22.581064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.580901    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61301b1b-bbd6-4a84-a409-055694ddc1b5-tls-certs\") pod \"61301b1b-bbd6-4a84-a409-055694ddc1b5\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") "
Jul 06 21:08:22.581064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.580931    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-home\") pod \"61301b1b-bbd6-4a84-a409-055694ddc1b5\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") "
Jul 06 21:08:22.581064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.580955    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-tmp-dir\") pod \"61301b1b-bbd6-4a84-a409-055694ddc1b5\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") "
Jul 06 21:08:22.581064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.581042    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-model-cache\") pod \"61301b1b-bbd6-4a84-a409-055694ddc1b5\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") "
Jul 06 21:08:22.581064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.581068    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-dshm\") pod \"61301b1b-bbd6-4a84-a409-055694ddc1b5\" (UID: \"61301b1b-bbd6-4a84-a409-055694ddc1b5\") "
Jul 06 21:08:22.581639 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.581609    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "61301b1b-bbd6-4a84-a409-055694ddc1b5" (UID: "61301b1b-bbd6-4a84-a409-055694ddc1b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.582187 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.582157    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-home" (OuterVolumeSpecName: "home") pod "61301b1b-bbd6-4a84-a409-055694ddc1b5" (UID: "61301b1b-bbd6-4a84-a409-055694ddc1b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.584077 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.584036    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61301b1b-bbd6-4a84-a409-055694ddc1b5-kube-api-access-5mwp7" (OuterVolumeSpecName: "kube-api-access-5mwp7") pod "61301b1b-bbd6-4a84-a409-055694ddc1b5" (UID: "61301b1b-bbd6-4a84-a409-055694ddc1b5"). InnerVolumeSpecName "kube-api-access-5mwp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:08:22.584191 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.584090    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-dshm" (OuterVolumeSpecName: "dshm") pod "61301b1b-bbd6-4a84-a409-055694ddc1b5" (UID: "61301b1b-bbd6-4a84-a409-055694ddc1b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.584410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.584392    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e" (OuterVolumeSpecName: "kserve-pvc-source") pod "61301b1b-bbd6-4a84-a409-055694ddc1b5" (UID: "61301b1b-bbd6-4a84-a409-055694ddc1b5"). InnerVolumeSpecName "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:08:22.585409 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.585391    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61301b1b-bbd6-4a84-a409-055694ddc1b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61301b1b-bbd6-4a84-a409-055694ddc1b5" (UID: "61301b1b-bbd6-4a84-a409-055694ddc1b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:08:22.593371 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.593347    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61301b1b-bbd6-4a84-a409-055694ddc1b5" (UID: "61301b1b-bbd6-4a84-a409-055694ddc1b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.616736 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.616713    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:08:22.682153 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682118    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-tmp-dir\") pod \"e10f851c-fa67-48cc-b51a-b497e02561da\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") "
Jul 06 21:08:22.682352 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-home\") pod \"e10f851c-fa67-48cc-b51a-b497e02561da\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") "
Jul 06 21:08:22.682352 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682254    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-dshm\") pod \"e10f851c-fa67-48cc-b51a-b497e02561da\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") "
Jul 06 21:08:22.682352 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682283    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-model-cache\") pod \"e10f851c-fa67-48cc-b51a-b497e02561da\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") "
Jul 06 21:08:22.682539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682400    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"e10f851c-fa67-48cc-b51a-b497e02561da\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") "
Jul 06 21:08:22.682539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fvtq\" (UniqueName: \"kubernetes.io/projected/e10f851c-fa67-48cc-b51a-b497e02561da-kube-api-access-4fvtq\") pod \"e10f851c-fa67-48cc-b51a-b497e02561da\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") "
Jul 06 21:08:22.682539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f851c-fa67-48cc-b51a-b497e02561da-tls-certs\") pod \"e10f851c-fa67-48cc-b51a-b497e02561da\" (UID: \"e10f851c-fa67-48cc-b51a-b497e02561da\") "
Jul 06 21:08:22.682677 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682600    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-model-cache" (OuterVolumeSpecName: "model-cache") pod "e10f851c-fa67-48cc-b51a-b497e02561da" (UID: "e10f851c-fa67-48cc-b51a-b497e02561da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.682819 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682792    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-home" (OuterVolumeSpecName: "home") pod "e10f851c-fa67-48cc-b51a-b497e02561da" (UID: "e10f851c-fa67-48cc-b51a-b497e02561da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.682917 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682889    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mwp7\" (UniqueName: \"kubernetes.io/projected/61301b1b-bbd6-4a84-a409-055694ddc1b5-kube-api-access-5mwp7\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.682917 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682917    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61301b1b-bbd6-4a84-a409-055694ddc1b5-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.683125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682935    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.683125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682947    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.683125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682957    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.683125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682987    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.683125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.682997    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61301b1b-bbd6-4a84-a409-055694ddc1b5-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.683125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.683007    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.684886 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.684865    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10f851c-fa67-48cc-b51a-b497e02561da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e10f851c-fa67-48cc-b51a-b497e02561da" (UID: "e10f851c-fa67-48cc-b51a-b497e02561da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:08:22.685146 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.685127    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-dshm" (OuterVolumeSpecName: "dshm") pod "e10f851c-fa67-48cc-b51a-b497e02561da" (UID: "e10f851c-fa67-48cc-b51a-b497e02561da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.685555 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.685529    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10f851c-fa67-48cc-b51a-b497e02561da-kube-api-access-4fvtq" (OuterVolumeSpecName: "kube-api-access-4fvtq") pod "e10f851c-fa67-48cc-b51a-b497e02561da" (UID: "e10f851c-fa67-48cc-b51a-b497e02561da"). InnerVolumeSpecName "kube-api-access-4fvtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:08:22.685632 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.685560    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e" (OuterVolumeSpecName: "kserve-pvc-source") pod "e10f851c-fa67-48cc-b51a-b497e02561da" (UID: "e10f851c-fa67-48cc-b51a-b497e02561da"). InnerVolumeSpecName "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:08:22.694712 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.694689    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e10f851c-fa67-48cc-b51a-b497e02561da" (UID: "e10f851c-fa67-48cc-b51a-b497e02561da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:22.783896 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.783810    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fvtq\" (UniqueName: \"kubernetes.io/projected/e10f851c-fa67-48cc-b51a-b497e02561da-kube-api-access-4fvtq\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.783896 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.783840    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f851c-fa67-48cc-b51a-b497e02561da-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.783896 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.783851    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.783896 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.783858    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f851c-fa67-48cc-b51a-b497e02561da-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:22.783896 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.783887    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" "
Jul 06 21:08:22.812468 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.812448    2582 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e") on node "ip-10-0-131-57.ec2.internal"
Jul 06 21:08:22.885062 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.885022    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\""
Jul 06 21:08:22.984722 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.984677    2582 generic.go:356] "Generic (PLEG): container finished" podID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerID="2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad" exitCode=0
Jul 06 21:08:22.984722 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.984714    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"
Jul 06 21:08:22.985001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.984752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" event={"ID":"61301b1b-bbd6-4a84-a409-055694ddc1b5","Type":"ContainerDied","Data":"2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad"}
Jul 06 21:08:22.985001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.984804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw" event={"ID":"61301b1b-bbd6-4a84-a409-055694ddc1b5","Type":"ContainerDied","Data":"daa2262a2645e927fe84b4bdf5fdd40a80f53349cd308604f2c558ac219dcd50"}
Jul 06 21:08:22.985001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.984826    2582 scope.go:117] "RemoveContainer" containerID="2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad"
Jul 06 21:08:22.986819 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.986795    2582 generic.go:356] "Generic (PLEG): container finished" podID="e10f851c-fa67-48cc-b51a-b497e02561da" containerID="087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8" exitCode=0
Jul 06 21:08:22.986926 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.986820    2582 generic.go:356] "Generic (PLEG): container finished" podID="e10f851c-fa67-48cc-b51a-b497e02561da" containerID="db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a" exitCode=0
Jul 06 21:08:22.986926 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.986854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" event={"ID":"e10f851c-fa67-48cc-b51a-b497e02561da","Type":"ContainerDied","Data":"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"}
Jul 06 21:08:22.986926 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.986865    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"
Jul 06 21:08:22.986926 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.986900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" event={"ID":"e10f851c-fa67-48cc-b51a-b497e02561da","Type":"ContainerDied","Data":"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"}
Jul 06 21:08:22.987202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.986928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv" event={"ID":"e10f851c-fa67-48cc-b51a-b497e02561da","Type":"ContainerDied","Data":"da4bdc622dac91f6ff579a043d5246bbc9e4b65e7e14b362548714a2df0a7ae8"}
Jul 06 21:08:22.997785 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.997768    2582 scope.go:117] "RemoveContainer" containerID="2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad"
Jul 06 21:08:22.998086 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:08:22.998064    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad\": container with ID starting with 2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad not found: ID does not exist" containerID="2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad"
Jul 06 21:08:22.998179 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.998097    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad"} err="failed to get container status \"2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad\": rpc error: code = NotFound desc = could not find container \"2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad\": container with ID starting with 2a74c11c2bb67f91421013d810ebfe8a33ae342533e38bae7658924f9a6dd0ad not found: ID does not exist"
Jul 06 21:08:22.998179 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:22.998121    2582 scope.go:117] "RemoveContainer" containerID="087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"
Jul 06 21:08:23.010138 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.010099    2582 scope.go:117] "RemoveContainer" containerID="db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"
Jul 06 21:08:23.014343 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.014316    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"]
Jul 06 21:08:23.016904 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.016886    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw"]
Jul 06 21:08:23.020113 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.020096    2582 scope.go:117] "RemoveContainer" containerID="087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"
Jul 06 21:08:23.020361 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:08:23.020341    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8\": container with ID starting with 087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8 not found: ID does not exist" containerID="087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"
Jul 06 21:08:23.020447 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.020373    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"} err="failed to get container status \"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8\": rpc error: code = NotFound desc = could not find container \"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8\": container with ID starting with 087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8 not found: ID does not exist"
Jul 06 21:08:23.020447 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.020396    2582 scope.go:117] "RemoveContainer" containerID="db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"
Jul 06 21:08:23.020667 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:08:23.020652    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a\": container with ID starting with db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a not found: ID does not exist" containerID="db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"
Jul 06 21:08:23.020733 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.020673    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"} err="failed to get container status \"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a\": rpc error: code = NotFound desc = could not find container \"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a\": container with ID starting with db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a not found: ID does not exist"
Jul 06 21:08:23.020733 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.020694    2582 scope.go:117] "RemoveContainer" containerID="087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"
Jul 06 21:08:23.020905 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.020888    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8"} err="failed to get container status \"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8\": rpc error: code = NotFound desc = could not find container \"087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8\": container with ID starting with 087d39a215cc2e2e8df64485467c6aec236d8d6deef79e3ebf35037d2dc154e8 not found: ID does not exist"
Jul 06 21:08:23.021001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.020907    2582 scope.go:117] "RemoveContainer" containerID="db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"
Jul 06 21:08:23.021178 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.021162    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a"} err="failed to get container status \"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a\": rpc error: code = NotFound desc = could not find container \"db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a\": container with ID starting with db711edaa4141a094e65c10ddcfc8a00aa597bba2ac7ba68af42bd5dc88f268a not found: ID does not exist"
Jul 06 21:08:23.030604 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.030578    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"]
Jul 06 21:08:23.034776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.034722    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv"]
Jul 06 21:08:23.338568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.338485    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" path="/var/lib/kubelet/pods/61301b1b-bbd6-4a84-a409-055694ddc1b5/volumes"
Jul 06 21:08:23.338923 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:23.338901    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" path="/var/lib/kubelet/pods/e10f851c-fa67-48cc-b51a-b497e02561da/volumes"
Jul 06 21:08:26.217511 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:26.217385    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:26.227330 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:26.227299    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"]
Jul 06 21:08:26.306832 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:26.306802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:35.045102 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.045067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:35.241378 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.241350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:35.342829 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.342711    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:35.387029 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.386995    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:08:35.387704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387607    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="llm-d-routing-sidecar"
Jul 06 21:08:35.387704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387629    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="llm-d-routing-sidecar"
Jul 06 21:08:35.387704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387647    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main"
Jul 06 21:08:35.387704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387653    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main"
Jul 06 21:08:35.387704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387693    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main"
Jul 06 21:08:35.387704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387703    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main"
Jul 06 21:08:35.388073 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387828    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="61301b1b-bbd6-4a84-a409-055694ddc1b5" containerName="main"
Jul 06 21:08:35.388073 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387841    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="main"
Jul 06 21:08:35.388073 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.387850    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10f851c-fa67-48cc-b51a-b497e02561da" containerName="llm-d-routing-sidecar"
Jul 06 21:08:35.393657 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.393636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.398003 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.397984    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 06 21:08:35.398099 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.398006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s2r78\""
Jul 06 21:08:35.408794 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.408767    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:08:35.447155 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.447122    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:08:35.450573 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.450555    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.467189 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.467163    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:08:35.505373 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.505343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.505539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.505392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.505539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.505507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.505646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.505551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.505646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.505585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8cj\" (UniqueName: \"kubernetes.io/projected/7c320a63-64e7-443e-8af3-d974eb1485bd-kube-api-access-zk8cj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.505646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.505612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c320a63-64e7-443e-8af3-d974eb1485bd-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.581333 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.581276    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:35.606341 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.606341 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.606542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.606542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.606542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.606542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.606542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8cj\" (UniqueName: \"kubernetes.io/projected/7c320a63-64e7-443e-8af3-d974eb1485bd-kube-api-access-zk8cj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.606787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c320a63-64e7-443e-8af3-d974eb1485bd-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.606787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdxts\" (UniqueName: \"kubernetes.io/projected/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-kube-api-access-vdxts\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.606787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.606787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606654    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.606787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606686    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.606787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.606787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.606786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.607086 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.607015    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.608643 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.608609    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.608821 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.608802    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c320a63-64e7-443e-8af3-d974eb1485bd-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.618601 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.618573    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8cj\" (UniqueName: \"kubernetes.io/projected/7c320a63-64e7-443e-8af3-d974eb1485bd-kube-api-access-zk8cj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:35.707793 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.707756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708028 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.707803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdxts\" (UniqueName: \"kubernetes.io/projected/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-kube-api-access-vdxts\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708028 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.707832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708028 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.707887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708028 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.708018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708298 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.708095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708298 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.708204    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708298 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.708262    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.708413 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.708296    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.710169 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.710146    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.710362 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.710345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:35.716317 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:35.716293    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdxts\" (UniqueName: \"kubernetes.io/projected/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-kube-api-access-vdxts\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:37.831219 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:37.831174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:37.838250 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:37.838223    2582 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:37.932144 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:37.932106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:37.932328 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:37.932264    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:37.936278 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:37.936257    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:38.019981 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.019938    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/95d8fd9e3a89fba13ba92fa30ad2ea3eca1fe57a3adce27e13a21899d478620a/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:38.032649 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.032620    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:38.104076 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.103988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:38.134711 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.134673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:38.134873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.134814    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:38.139531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.139503    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:38.152096 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.152068    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:38.160185 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.160156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:08:38.257974 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.257940    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:08:38.258840 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:08:38.258817    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c320a63_64e7_443e_8af3_d974eb1485bd.slice/crio-9304c38fd690012d977761727c7e17c4b90fdf5726c1cb20d44c7362917b8d29 WatchSource:0}: Error finding container 9304c38fd690012d977761727c7e17c4b90fdf5726c1cb20d44c7362917b8d29: Status 404 returned error can't find the container with id 9304c38fd690012d977761727c7e17c4b90fdf5726c1cb20d44c7362917b8d29
Jul 06 21:08:38.324410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:38.324387    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:08:38.325283 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:08:38.325259    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522dcb8b_0790_4fc9_8224_14f0e1dadbd2.slice/crio-e5474f874bd81cd2f7f26137eefc534fc6a618dcd0abcb817f24702c779aab24 WatchSource:0}: Error finding container e5474f874bd81cd2f7f26137eefc534fc6a618dcd0abcb817f24702c779aab24: Status 404 returned error can't find the container with id e5474f874bd81cd2f7f26137eefc534fc6a618dcd0abcb817f24702c779aab24
Jul 06 21:08:39.085714 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:39.085677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7c320a63-64e7-443e-8af3-d974eb1485bd","Type":"ContainerStarted","Data":"afef6e655f6f4fb8dca4f16fc90371ab43459669544d6b2607c5cdb5409ece86"}
Jul 06 21:08:39.086112 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:39.085742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7c320a63-64e7-443e-8af3-d974eb1485bd","Type":"ContainerStarted","Data":"9304c38fd690012d977761727c7e17c4b90fdf5726c1cb20d44c7362917b8d29"}
Jul 06 21:08:39.087652 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:39.087624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"522dcb8b-0790-4fc9-8224-14f0e1dadbd2","Type":"ContainerStarted","Data":"ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428"}
Jul 06 21:08:39.087774 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:39.087671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"522dcb8b-0790-4fc9-8224-14f0e1dadbd2","Type":"ContainerStarted","Data":"e5474f874bd81cd2f7f26137eefc534fc6a618dcd0abcb817f24702c779aab24"}
Jul 06 21:08:39.104920 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:39.104890    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:39.112734 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:39.112644    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=4.112595085 podStartE2EDuration="4.112595085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:39.109274299 +0000 UTC m=+2016.443614921" watchObservedRunningTime="2026-07-06 21:08:39.112595085 +0000 UTC m=+2016.446935683"
Jul 06 21:08:39.135121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:39.135059    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=4.135037804 podStartE2EDuration="4.135037804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:39.132001775 +0000 UTC m=+2016.466342369" watchObservedRunningTime="2026-07-06 21:08:39.135037804 +0000 UTC m=+2016.469378402"
Jul 06 21:08:41.261284 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:41.261224    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" containerID="cri-o://d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99" gracePeriod=45
Jul 06 21:08:41.887404 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:41.887363    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 21:08:42.042262 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.042235    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:08:42.123461 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.123363    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7ca33fd-976f-433d-81f1-9e9213134248" containerID="d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99" exitCode=0
Jul 06 21:08:42.123754 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.123442    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"
Jul 06 21:08:42.123940 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.123453    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" event={"ID":"a7ca33fd-976f-433d-81f1-9e9213134248","Type":"ContainerDied","Data":"d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99"}
Jul 06 21:08:42.124213 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.124186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt" event={"ID":"a7ca33fd-976f-433d-81f1-9e9213134248","Type":"ContainerDied","Data":"09cf798507ebd479216204c4ed31d17cf0c3579e0d27696ebd16f2e1a2e507f4"}
Jul 06 21:08:42.124356 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.124253    2582 scope.go:117] "RemoveContainer" containerID="d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99"
Jul 06 21:08:42.137047 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.137027    2582 scope.go:117] "RemoveContainer" containerID="f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885"
Jul 06 21:08:42.148088 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.148067    2582 scope.go:117] "RemoveContainer" containerID="d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99"
Jul 06 21:08:42.148428 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:08:42.148390    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99\": container with ID starting with d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99 not found: ID does not exist" containerID="d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99"
Jul 06 21:08:42.148526 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.148431    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99"} err="failed to get container status \"d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99\": rpc error: code = NotFound desc = could not find container \"d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99\": container with ID starting with d04e46728a4cbb46e85706cc565134d63e05e06f605855f1e5c68cbd91baff99 not found: ID does not exist"
Jul 06 21:08:42.148526 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.148458    2582 scope.go:117] "RemoveContainer" containerID="f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885"
Jul 06 21:08:42.148764 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:08:42.148741    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885\": container with ID starting with f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885 not found: ID does not exist" containerID="f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885"
Jul 06 21:08:42.148821 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.148772    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885"} err="failed to get container status \"f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885\": rpc error: code = NotFound desc = could not find container \"f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885\": container with ID starting with f4716480599f820c3d9408cf7101bc077fec524553be1b951d4929413bb43885 not found: ID does not exist"
Jul 06 21:08:42.180432 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.180398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-dshm\") pod \"a7ca33fd-976f-433d-81f1-9e9213134248\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") "
Jul 06 21:08:42.180623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.180446    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-home\") pod \"a7ca33fd-976f-433d-81f1-9e9213134248\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") "
Jul 06 21:08:42.180623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.180476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-tmp-dir\") pod \"a7ca33fd-976f-433d-81f1-9e9213134248\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") "
Jul 06 21:08:42.180623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.180559    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-kserve-provision-location\") pod \"a7ca33fd-976f-433d-81f1-9e9213134248\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") "
Jul 06 21:08:42.180623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.180586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-model-cache\") pod \"a7ca33fd-976f-433d-81f1-9e9213134248\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") "
Jul 06 21:08:42.180848 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.180677    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ca33fd-976f-433d-81f1-9e9213134248-tls-certs\") pod \"a7ca33fd-976f-433d-81f1-9e9213134248\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") "
Jul 06 21:08:42.180848 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.180719    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrnpx\" (UniqueName: \"kubernetes.io/projected/a7ca33fd-976f-433d-81f1-9e9213134248-kube-api-access-wrnpx\") pod \"a7ca33fd-976f-433d-81f1-9e9213134248\" (UID: \"a7ca33fd-976f-433d-81f1-9e9213134248\") "
Jul 06 21:08:42.181036 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.181011    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7ca33fd-976f-433d-81f1-9e9213134248" (UID: "a7ca33fd-976f-433d-81f1-9e9213134248"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:42.183079 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.181144    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:42.183079 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.181575    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-home" (OuterVolumeSpecName: "home") pod "a7ca33fd-976f-433d-81f1-9e9213134248" (UID: "a7ca33fd-976f-433d-81f1-9e9213134248"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:42.183455 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.183430    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ca33fd-976f-433d-81f1-9e9213134248-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7ca33fd-976f-433d-81f1-9e9213134248" (UID: "a7ca33fd-976f-433d-81f1-9e9213134248"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:08:42.183950 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.183922    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ca33fd-976f-433d-81f1-9e9213134248-kube-api-access-wrnpx" (OuterVolumeSpecName: "kube-api-access-wrnpx") pod "a7ca33fd-976f-433d-81f1-9e9213134248" (UID: "a7ca33fd-976f-433d-81f1-9e9213134248"). InnerVolumeSpecName "kube-api-access-wrnpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:08:42.184073 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.183910    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-dshm" (OuterVolumeSpecName: "dshm") pod "a7ca33fd-976f-433d-81f1-9e9213134248" (UID: "a7ca33fd-976f-433d-81f1-9e9213134248"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:42.200718 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.200674    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7ca33fd-976f-433d-81f1-9e9213134248" (UID: "a7ca33fd-976f-433d-81f1-9e9213134248"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:42.217293 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.217245    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7ca33fd-976f-433d-81f1-9e9213134248" (UID: "a7ca33fd-976f-433d-81f1-9e9213134248"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:08:42.281852 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.281815    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnpx\" (UniqueName: \"kubernetes.io/projected/a7ca33fd-976f-433d-81f1-9e9213134248-kube-api-access-wrnpx\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:42.281852 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.281850    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:42.281852 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.281863    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:42.282477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.281901    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:42.282477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.281914    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ca33fd-976f-433d-81f1-9e9213134248-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:42.282477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.281927    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ca33fd-976f-433d-81f1-9e9213134248-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:08:42.457669 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.457625    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"]
Jul 06 21:08:42.461745 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:42.461719    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt"]
Jul 06 21:08:43.337606 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:43.337567    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" path="/var/lib/kubelet/pods/a7ca33fd-976f-433d-81f1-9e9213134248/volumes"
Jul 06 21:08:43.724222 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:43.724184    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:48.105035 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:48.104986    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:08:48.106459 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:48.106431    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:08:50.952168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:50.952130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:53.355708 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.355672    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"]
Jul 06 21:08:53.356368 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.356346    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main"
Jul 06 21:08:53.356412 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.356374    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main"
Jul 06 21:08:53.356412 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.356395    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="storage-initializer"
Jul 06 21:08:53.356412 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.356406    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="storage-initializer"
Jul 06 21:08:53.356526 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.356514    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ca33fd-976f-433d-81f1-9e9213134248" containerName="main"
Jul 06 21:08:53.362088 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.362062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.365627 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.365601    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:08:53.366612 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.366586    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 06 21:08:53.380645 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.380553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"]
Jul 06 21:08:53.423166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.423125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:53.506140 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.506108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-home\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.506140 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.506143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-model-cache\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.506353 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.506176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-dshm\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.506353 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.506255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tls-certs\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.506353 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.506326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.506463 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.506375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqp8\" (UniqueName: \"kubernetes.io/projected/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kube-api-access-ftqp8\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.506463 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.506402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tmp-dir\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.607996 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.607885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-home\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.607996 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.607925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-model-cache\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.607996 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.607947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-dshm\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tls-certs\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqp8\" (UniqueName: \"kubernetes.io/projected/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kube-api-access-ftqp8\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tmp-dir\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608516 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608377    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-home\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608516 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608502    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608612 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608519    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tmp-dir\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.608663 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.608619    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-model-cache\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.610863 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.610840    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-dshm\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.611273 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.611249    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tls-certs\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.617441 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.617417    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqp8\" (UniqueName: \"kubernetes.io/projected/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kube-api-access-ftqp8\") pod \"custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.686575 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.686534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:08:53.734582 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.734540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:08:53.870314 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:53.870280    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"]
Jul 06 21:08:53.873179 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:08:53.873137    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9087cd_a365_4464_bfa3_8cacf5d3b4e5.slice/crio-60452a3349a5f2f6aec384e708516fc2d157b2d2c72ef92657c278947801adfe WatchSource:0}: Error finding container 60452a3349a5f2f6aec384e708516fc2d157b2d2c72ef92657c278947801adfe: Status 404 returned error can't find the container with id 60452a3349a5f2f6aec384e708516fc2d157b2d2c72ef92657c278947801adfe
Jul 06 21:08:54.216742 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:54.216487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" event={"ID":"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5","Type":"ContainerStarted","Data":"37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a"}
Jul 06 21:08:54.216742 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:54.216562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" event={"ID":"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5","Type":"ContainerStarted","Data":"60452a3349a5f2f6aec384e708516fc2d157b2d2c72ef92657c278947801adfe"}
Jul 06 21:08:58.105525 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:58.105479    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:08:58.252739 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:58.252703    2582 generic.go:356] "Generic (PLEG): container finished" podID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerID="37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a" exitCode=0
Jul 06 21:08:58.252927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:58.252783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" event={"ID":"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5","Type":"ContainerDied","Data":"37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a"}
Jul 06 21:08:59.263325 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:59.263286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" event={"ID":"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5","Type":"ContainerStarted","Data":"bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e"}
Jul 06 21:08:59.296233 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:59.296160    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podStartSLOduration=2.234367002 podStartE2EDuration="6.296136824s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.061769822s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:59.290719548 +0000 UTC m=+2036.625060145" watchObservedRunningTime="2026-07-06 21:08:59.296136824 +0000 UTC m=+2036.630477421"
Jul 06 21:08:59.687485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:08:59.687437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:09:03.061283 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:03.061234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:09:03.687253 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:03.687212    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:09:03.688742 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:03.688708    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:09:08.104979 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:08.104908    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:09:13.687641 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:13.687594    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:09:18.105491 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:18.105431    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:09:23.687752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:23.687702    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:09:28.105486 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:28.105371    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:09:33.687680 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:33.687607    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:09:38.105208 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:38.105162    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:09:43.687206 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:43.687156    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:09:48.104528 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:48.104475    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:09:53.687834 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:53.687771    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:09:58.104861 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:09:58.104816    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:10:03.368221 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:03.368187    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:10:03.374820 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:03.374779    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:10:03.687320 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:03.687269    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:10:08.104674 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:08.104628    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:10:13.687555 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:13.687508    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:10:18.105214 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:18.105173    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:10:23.687536 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:23.687482    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:10:28.105185 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:28.105140    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:10:33.687929 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:33.687877    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:10:38.105088 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:38.105038    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:10:43.687684 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:43.687639    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:10:48.104437 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:48.104392    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:10:53.687656 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:53.687607    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:10:58.104728 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:58.104643    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:10:59.459885 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:10:59.459851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:03.688064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:03.688008    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:11:08.104890 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:08.104843    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:11:09.881978 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:09.881933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:13.687950 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:13.687899    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:11:18.104646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:18.104605    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:11:23.687553 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:23.687505    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:11:28.114109 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:28.114070    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:11:28.122159 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:28.122134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:11:33.687015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:33.686976    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 06 21:11:43.697319 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:43.697285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:11:43.705505 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:43.705477    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:11:48.932557 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:48.932522    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"]
Jul 06 21:11:48.934728 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:48.934701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:49.028190 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:49.028144    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:54.856676 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:54.856636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:11:54.886176 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:54.886139    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:11:54.931410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:11:54.931381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:03.958017 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:03.957940    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" containerID="cri-o://bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e" gracePeriod=45
Jul 06 21:12:04.720556 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.720530    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:12:04.798160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798078    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-model-cache\") pod \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") "
Jul 06 21:12:04.798160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tls-certs\") pod \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") "
Jul 06 21:12:04.798361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798161    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-dshm\") pod \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") "
Jul 06 21:12:04.798361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798215    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftqp8\" (UniqueName: \"kubernetes.io/projected/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kube-api-access-ftqp8\") pod \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") "
Jul 06 21:12:04.798361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798268    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tmp-dir\") pod \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") "
Jul 06 21:12:04.798361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-home\") pod \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") "
Jul 06 21:12:04.798361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kserve-provision-location\") pod \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\" (UID: \"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5\") "
Jul 06 21:12:04.799060 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798414    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" (UID: "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:04.799060 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.798708    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:04.799149 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.799061    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-home" (OuterVolumeSpecName: "home") pod "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" (UID: "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:04.801009 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.800957    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kube-api-access-ftqp8" (OuterVolumeSpecName: "kube-api-access-ftqp8") pod "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" (UID: "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5"). InnerVolumeSpecName "kube-api-access-ftqp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:12:04.801119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.800951    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" (UID: "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:12:04.801620 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.801603    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-dshm" (OuterVolumeSpecName: "dshm") pod "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" (UID: "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:04.810755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.810705    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" (UID: "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:04.839268 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.839225    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" (UID: "3c9087cd-a365-4464-bfa3-8cacf5d3b4e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:04.899398 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.899361    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:04.899398 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.899395    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:04.899398 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.899405    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftqp8\" (UniqueName: \"kubernetes.io/projected/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kube-api-access-ftqp8\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:04.899598 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.899415    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:04.899598 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.899429    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:04.899598 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:04.899439    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:05.517923 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.517882    2582 generic.go:356] "Generic (PLEG): container finished" podID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerID="bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e" exitCode=0
Jul 06 21:12:05.518387 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.517945    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"
Jul 06 21:12:05.518387 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.517990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" event={"ID":"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5","Type":"ContainerDied","Data":"bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e"}
Jul 06 21:12:05.518387 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.518053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" event={"ID":"3c9087cd-a365-4464-bfa3-8cacf5d3b4e5","Type":"ContainerDied","Data":"60452a3349a5f2f6aec384e708516fc2d157b2d2c72ef92657c278947801adfe"}
Jul 06 21:12:05.518387 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.518084    2582 scope.go:117] "RemoveContainer" containerID="bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e"
Jul 06 21:12:05.529767 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.529735    2582 scope.go:117] "RemoveContainer" containerID="37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a"
Jul 06 21:12:05.538678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.538629    2582 scope.go:117] "RemoveContainer" containerID="bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e"
Jul 06 21:12:05.538944 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:12:05.538919    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e\": container with ID starting with bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e not found: ID does not exist" containerID="bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e"
Jul 06 21:12:05.539080 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.538953    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e"} err="failed to get container status \"bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e\": rpc error: code = NotFound desc = could not find container \"bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e\": container with ID starting with bd2b5403ebac5d48172c90b1374bac33639d71311382e803cbbbbab15972b89e not found: ID does not exist"
Jul 06 21:12:05.539080 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.539006    2582 scope.go:117] "RemoveContainer" containerID="37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a"
Jul 06 21:12:05.539328 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:12:05.539309    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a\": container with ID starting with 37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a not found: ID does not exist" containerID="37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a"
Jul 06 21:12:05.539469 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.539334    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a"} err="failed to get container status \"37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a\": rpc error: code = NotFound desc = could not find container \"37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a\": container with ID starting with 37d87a1f3290974a126672f78b8e9f6df104b515c1b48438f080b5a7d8b5dc2a not found: ID does not exist"
Jul 06 21:12:05.542112 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.542090    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"]
Jul 06 21:12:05.545477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.545455    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d"]
Jul 06 21:12:05.698070 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:05.698030    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:12:07.337689 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:07.337654    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" path="/var/lib/kubelet/pods/3c9087cd-a365-4464-bfa3-8cacf5d3b4e5/volumes"
Jul 06 21:12:09.910464 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:09.910422    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" containerID="cri-o://afef6e655f6f4fb8dca4f16fc90371ab43459669544d6b2607c5cdb5409ece86" gracePeriod=45
Jul 06 21:12:10.116030 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.115686    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 06 21:12:10.557754 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.557715    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerID="afef6e655f6f4fb8dca4f16fc90371ab43459669544d6b2607c5cdb5409ece86" exitCode=0
Jul 06 21:12:10.557996 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.557860    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7c320a63-64e7-443e-8af3-d974eb1485bd","Type":"ContainerDied","Data":"afef6e655f6f4fb8dca4f16fc90371ab43459669544d6b2607c5cdb5409ece86"}
Jul 06 21:12:10.727924 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.727894    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:12:10.758236 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.758204    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-home\") pod \"7c320a63-64e7-443e-8af3-d974eb1485bd\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") "
Jul 06 21:12:10.758431 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.758249    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c320a63-64e7-443e-8af3-d974eb1485bd-tls-certs\") pod \"7c320a63-64e7-443e-8af3-d974eb1485bd\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") "
Jul 06 21:12:10.758431 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.758404    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"7c320a63-64e7-443e-8af3-d974eb1485bd\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") "
Jul 06 21:12:10.758567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.758456    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-tmp-dir\") pod \"7c320a63-64e7-443e-8af3-d974eb1485bd\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") "
Jul 06 21:12:10.758567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.758473    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk8cj\" (UniqueName: \"kubernetes.io/projected/7c320a63-64e7-443e-8af3-d974eb1485bd-kube-api-access-zk8cj\") pod \"7c320a63-64e7-443e-8af3-d974eb1485bd\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") "
Jul 06 21:12:10.758567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.758499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-dshm\") pod \"7c320a63-64e7-443e-8af3-d974eb1485bd\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") "
Jul 06 21:12:10.758567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.758519    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-model-cache\") pod \"7c320a63-64e7-443e-8af3-d974eb1485bd\" (UID: \"7c320a63-64e7-443e-8af3-d974eb1485bd\") "
Jul 06 21:12:10.759104 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.759025    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c320a63-64e7-443e-8af3-d974eb1485bd" (UID: "7c320a63-64e7-443e-8af3-d974eb1485bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:10.759104 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.759038    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-home" (OuterVolumeSpecName: "home") pod "7c320a63-64e7-443e-8af3-d974eb1485bd" (UID: "7c320a63-64e7-443e-8af3-d974eb1485bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:10.761005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.760946    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c320a63-64e7-443e-8af3-d974eb1485bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c320a63-64e7-443e-8af3-d974eb1485bd" (UID: "7c320a63-64e7-443e-8af3-d974eb1485bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:12:10.761527 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.761505    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c320a63-64e7-443e-8af3-d974eb1485bd-kube-api-access-zk8cj" (OuterVolumeSpecName: "kube-api-access-zk8cj") pod "7c320a63-64e7-443e-8af3-d974eb1485bd" (UID: "7c320a63-64e7-443e-8af3-d974eb1485bd"). InnerVolumeSpecName "kube-api-access-zk8cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:12:10.761617 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.761535    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-dshm" (OuterVolumeSpecName: "dshm") pod "7c320a63-64e7-443e-8af3-d974eb1485bd" (UID: "7c320a63-64e7-443e-8af3-d974eb1485bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:10.762079 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.762056    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e" (OuterVolumeSpecName: "kserve-pvc-source") pod "7c320a63-64e7-443e-8af3-d974eb1485bd" (UID: "7c320a63-64e7-443e-8af3-d974eb1485bd"). InnerVolumeSpecName "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:12:10.777876 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.777844    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c320a63-64e7-443e-8af3-d974eb1485bd" (UID: "7c320a63-64e7-443e-8af3-d974eb1485bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:10.859932 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.859893    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:10.859932 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.859926    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8cj\" (UniqueName: \"kubernetes.io/projected/7c320a63-64e7-443e-8af3-d974eb1485bd-kube-api-access-zk8cj\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:10.859932 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.859938    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:10.859932 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.859946    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:10.860254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.859954    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c320a63-64e7-443e-8af3-d974eb1485bd-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:10.860254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:10.859981    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c320a63-64e7-443e-8af3-d974eb1485bd-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:11.564288 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:11.564256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7c320a63-64e7-443e-8af3-d974eb1485bd","Type":"ContainerDied","Data":"9304c38fd690012d977761727c7e17c4b90fdf5726c1cb20d44c7362917b8d29"}
Jul 06 21:12:11.564704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:11.564268    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 06 21:12:11.564704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:11.564318    2582 scope.go:117] "RemoveContainer" containerID="afef6e655f6f4fb8dca4f16fc90371ab43459669544d6b2607c5cdb5409ece86"
Jul 06 21:12:11.583117 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:11.583091    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:12:11.588910 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:11.588886    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 06 21:12:11.611524 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:11.611442    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:12:13.337632 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:13.337601    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" path="/var/lib/kubelet/pods/7c320a63-64e7-443e-8af3-d974eb1485bd/volumes"
Jul 06 21:12:16.597941 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.597902    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"]
Jul 06 21:12:16.598454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598371    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main"
Jul 06 21:12:16.598454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598393    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main"
Jul 06 21:12:16.598454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598411    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main"
Jul 06 21:12:16.598454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598418    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main"
Jul 06 21:12:16.598454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598432    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="storage-initializer"
Jul 06 21:12:16.598454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598437    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="storage-initializer"
Jul 06 21:12:16.598780 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598505    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c320a63-64e7-443e-8af3-d974eb1485bd" containerName="main"
Jul 06 21:12:16.598780 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.598513    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9087cd-a365-4464-bfa3-8cacf5d3b4e5" containerName="main"
Jul 06 21:12:16.603864 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.603837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.607545 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.607526    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 06 21:12:16.608094 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.608074    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:12:16.616146 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.616122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"]
Jul 06 21:12:16.649126 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.649090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:16.714323 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.714291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-kserve-provision-location\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.714515 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.714337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-model-cache\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.714515 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.714426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a0f86f-4290-419a-9913-6df2287757a9-tls-certs\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.714515 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.714485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-dshm\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.714683 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.714528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-home\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.714683 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.714552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9dnv\" (UniqueName: \"kubernetes.io/projected/53a0f86f-4290-419a-9913-6df2287757a9-kube-api-access-z9dnv\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.714683 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.714639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-tmp-dir\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.815940 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.815899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-tmp-dir\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.815985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-kserve-provision-location\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-model-cache\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a0f86f-4290-419a-9913-6df2287757a9-tls-certs\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-dshm\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-home\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816479 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9dnv\" (UniqueName: \"kubernetes.io/projected/53a0f86f-4290-419a-9913-6df2287757a9-kube-api-access-z9dnv\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816479 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816338    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-tmp-dir\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816479 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816405    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-kserve-provision-location\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816479 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816451    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-home\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.816647 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.816530    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-model-cache\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.818352 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.818327    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-dshm\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.818731 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.818708    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a0f86f-4290-419a-9913-6df2287757a9-tls-certs\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.825701 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.825678    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9dnv\" (UniqueName: \"kubernetes.io/projected/53a0f86f-4290-419a-9913-6df2287757a9-kube-api-access-z9dnv\") pod \"router-with-refs-test-kserve-5cf96c9bd7-bbfd9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:16.914867 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:16.914778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:17.001080 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:17.001042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:17.079389 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:17.079196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:17.081616 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:17.081587    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"]
Jul 06 21:12:17.083111 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:12:17.083088    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a0f86f_4290_419a_9913_6df2287757a9.slice/crio-7eb0304618d30c2e1f3853cc66ace8e7c1ef47eea56b94d4fd07ed7b703bc978 WatchSource:0}: Error finding container 7eb0304618d30c2e1f3853cc66ace8e7c1ef47eea56b94d4fd07ed7b703bc978: Status 404 returned error can't find the container with id 7eb0304618d30c2e1f3853cc66ace8e7c1ef47eea56b94d4fd07ed7b703bc978
Jul 06 21:12:17.600941 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:17.600898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" event={"ID":"53a0f86f-4290-419a-9913-6df2287757a9","Type":"ContainerStarted","Data":"ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c"}
Jul 06 21:12:17.601360 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:17.600976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" event={"ID":"53a0f86f-4290-419a-9913-6df2287757a9","Type":"ContainerStarted","Data":"7eb0304618d30c2e1f3853cc66ace8e7c1ef47eea56b94d4fd07ed7b703bc978"}
Jul 06 21:12:20.620525 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:20.620493    2582 generic.go:356] "Generic (PLEG): container finished" podID="53a0f86f-4290-419a-9913-6df2287757a9" containerID="ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c" exitCode=0
Jul 06 21:12:20.620812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:20.620530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" event={"ID":"53a0f86f-4290-419a-9913-6df2287757a9","Type":"ContainerDied","Data":"ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c"}
Jul 06 21:12:21.628310 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:21.628269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" event={"ID":"53a0f86f-4290-419a-9913-6df2287757a9","Type":"ContainerStarted","Data":"cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472"}
Jul 06 21:12:21.666989 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:21.660712    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podStartSLOduration=2.214874829 podStartE2EDuration="5.660691153s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.445816324s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:21.660123256 +0000 UTC m=+2238.994463852" watchObservedRunningTime="2026-07-06 21:12:21.660691153 +0000 UTC m=+2238.995031749"
Jul 06 21:12:21.915593 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:21.915491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:26.641520 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:26.641427    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="522dcb8b-0790-4fc9-8224-14f0e1dadbd2" containerName="main" containerID="cri-o://ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428" gracePeriod=45
Jul 06 21:12:26.915323 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:26.915226    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:12:26.917099 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:26.917069    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:12:27.413376 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.413352    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:12:27.525225 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525188    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-dshm\") pod \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") "
Jul 06 21:12:27.525403 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525281    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdxts\" (UniqueName: \"kubernetes.io/projected/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-kube-api-access-vdxts\") pod \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") "
Jul 06 21:12:27.525403 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525341    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tls-certs\") pod \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") "
Jul 06 21:12:27.525541 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525518    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tmp-dir\") pod \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") "
Jul 06 21:12:27.525611 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525570    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-model-cache\") pod \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") "
Jul 06 21:12:27.525709 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525691    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") pod \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") "
Jul 06 21:12:27.525760 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525739    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-home\") pod \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\" (UID: \"522dcb8b-0790-4fc9-8224-14f0e1dadbd2\") "
Jul 06 21:12:27.525833 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.525808    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "522dcb8b-0790-4fc9-8224-14f0e1dadbd2" (UID: "522dcb8b-0790-4fc9-8224-14f0e1dadbd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:27.526124 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.526104    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:27.526750 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.526712    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-home" (OuterVolumeSpecName: "home") pod "522dcb8b-0790-4fc9-8224-14f0e1dadbd2" (UID: "522dcb8b-0790-4fc9-8224-14f0e1dadbd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:27.527893 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.527862    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "522dcb8b-0790-4fc9-8224-14f0e1dadbd2" (UID: "522dcb8b-0790-4fc9-8224-14f0e1dadbd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:12:27.528035 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.527895    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-dshm" (OuterVolumeSpecName: "dshm") pod "522dcb8b-0790-4fc9-8224-14f0e1dadbd2" (UID: "522dcb8b-0790-4fc9-8224-14f0e1dadbd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:27.528035 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.527925    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-kube-api-access-vdxts" (OuterVolumeSpecName: "kube-api-access-vdxts") pod "522dcb8b-0790-4fc9-8224-14f0e1dadbd2" (UID: "522dcb8b-0790-4fc9-8224-14f0e1dadbd2"). InnerVolumeSpecName "kube-api-access-vdxts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:12:27.528998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.528976    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e" (OuterVolumeSpecName: "kserve-pvc-source") pod "522dcb8b-0790-4fc9-8224-14f0e1dadbd2" (UID: "522dcb8b-0790-4fc9-8224-14f0e1dadbd2"). InnerVolumeSpecName "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:12:27.544119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.544081    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "522dcb8b-0790-4fc9-8224-14f0e1dadbd2" (UID: "522dcb8b-0790-4fc9-8224-14f0e1dadbd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:12:27.627423 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.627380    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdxts\" (UniqueName: \"kubernetes.io/projected/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-kube-api-access-vdxts\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:27.627423 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.627410    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:27.627423 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.627421    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:27.627748 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.627449    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" "
Jul 06 21:12:27.627748 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.627457    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:27.627748 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.627469    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522dcb8b-0790-4fc9-8224-14f0e1dadbd2-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:12:27.655446 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.655423    2582 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e") on node "ip-10-0-131-57.ec2.internal"
Jul 06 21:12:27.664375 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.664345    2582 generic.go:356] "Generic (PLEG): container finished" podID="522dcb8b-0790-4fc9-8224-14f0e1dadbd2" containerID="ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428" exitCode=0
Jul 06 21:12:27.664542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.664393    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 06 21:12:27.664542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.664430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"522dcb8b-0790-4fc9-8224-14f0e1dadbd2","Type":"ContainerDied","Data":"ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428"}
Jul 06 21:12:27.664542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.664504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"522dcb8b-0790-4fc9-8224-14f0e1dadbd2","Type":"ContainerDied","Data":"e5474f874bd81cd2f7f26137eefc534fc6a618dcd0abcb817f24702c779aab24"}
Jul 06 21:12:27.664542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.664534    2582 scope.go:117] "RemoveContainer" containerID="ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428"
Jul 06 21:12:27.673994 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.673953    2582 scope.go:117] "RemoveContainer" containerID="ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428"
Jul 06 21:12:27.674273 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:12:27.674249    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428\": container with ID starting with ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428 not found: ID does not exist" containerID="ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428"
Jul 06 21:12:27.674344 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.674279    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428"} err="failed to get container status \"ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428\": rpc error: code = NotFound desc = could not find container \"ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428\": container with ID starting with ee999826d0ec9d5dae6e7afbc05203827e289ec3665995484b57000b93be9428 not found: ID does not exist"
Jul 06 21:12:27.695927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.695899    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:12:27.704167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.704140    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 06 21:12:27.728143 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:27.728116    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-cff970c0-1328-4734-8bb0-45cf29b3a727\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bdf47d53e9b33a8e\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"csi-4f8c790e17a2438de026e49e8530a559a132ebc946888f44010b712080eb10db\""
Jul 06 21:12:29.345356 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:29.345310    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522dcb8b-0790-4fc9-8224-14f0e1dadbd2" path="/var/lib/kubelet/pods/522dcb8b-0790-4fc9-8224-14f0e1dadbd2/volumes"
Jul 06 21:12:36.915803 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:36.915752    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:12:38.693987 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.693923    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"]
Jul 06 21:12:38.694650 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.694620    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522dcb8b-0790-4fc9-8224-14f0e1dadbd2" containerName="main"
Jul 06 21:12:38.694650 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.694647    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="522dcb8b-0790-4fc9-8224-14f0e1dadbd2" containerName="main"
Jul 06 21:12:38.694829 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.694771    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="522dcb8b-0790-4fc9-8224-14f0e1dadbd2" containerName="main"
Jul 06 21:12:38.697505 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.697485    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.700481 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.700463    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 06 21:12:38.715663 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.715635    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"]
Jul 06 21:12:38.715813 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.715680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:38.734873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.734843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-tmp-dir\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.735072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.734904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-kserve-provision-location\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.735072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.734941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b556f97e-f89b-441a-8f09-8654d0088bfa-tls-certs\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.735072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.734992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-home\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.735072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.735022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-model-cache\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.735072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.735068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9gs\" (UniqueName: \"kubernetes.io/projected/b556f97e-f89b-441a-8f09-8654d0088bfa-kube-api-access-8w9gs\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.735273 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.735092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-dshm\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.835846 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.835787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9gs\" (UniqueName: \"kubernetes.io/projected/b556f97e-f89b-441a-8f09-8654d0088bfa-kube-api-access-8w9gs\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836086 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.835863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-dshm\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836086 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.835952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-tmp-dir\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836086 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-kserve-provision-location\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836278 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b556f97e-f89b-441a-8f09-8654d0088bfa-tls-certs\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836278 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-home\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836278 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-model-cache\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836462 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836437    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-tmp-dir\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836526 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-home\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836592 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836573    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-kserve-provision-location\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.836640 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.836626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-model-cache\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.838785 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.838754    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-dshm\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.839003 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.838986    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b556f97e-f89b-441a-8f09-8654d0088bfa-tls-certs\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:38.845067 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:38.845040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9gs\" (UniqueName: \"kubernetes.io/projected/b556f97e-f89b-441a-8f09-8654d0088bfa-kube-api-access-8w9gs\") pod \"stop-feature-test-kserve-9b9d66cf-2xd45\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:39.006453 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:39.006367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:39.008386 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:39.008363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:39.169930 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:39.169902    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"]
Jul 06 21:12:39.171233 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:12:39.171203    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb556f97e_f89b_441a_8f09_8654d0088bfa.slice/crio-1320b07a9af58c7d97b7339d4f6bae935162a13782bea2429fe935d7a63ad97d WatchSource:0}: Error finding container 1320b07a9af58c7d97b7339d4f6bae935162a13782bea2429fe935d7a63ad97d: Status 404 returned error can't find the container with id 1320b07a9af58c7d97b7339d4f6bae935162a13782bea2429fe935d7a63ad97d
Jul 06 21:12:39.740830 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:39.740792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" event={"ID":"b556f97e-f89b-441a-8f09-8654d0088bfa","Type":"ContainerStarted","Data":"87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887"}
Jul 06 21:12:39.741340 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:39.740848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" event={"ID":"b556f97e-f89b-441a-8f09-8654d0088bfa","Type":"ContainerStarted","Data":"1320b07a9af58c7d97b7339d4f6bae935162a13782bea2429fe935d7a63ad97d"}
Jul 06 21:12:45.784875 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:45.784837    2582 generic.go:356] "Generic (PLEG): container finished" podID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerID="87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887" exitCode=0
Jul 06 21:12:45.785390 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:45.784909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" event={"ID":"b556f97e-f89b-441a-8f09-8654d0088bfa","Type":"ContainerDied","Data":"87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887"}
Jul 06 21:12:46.794025 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:46.793991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" event={"ID":"b556f97e-f89b-441a-8f09-8654d0088bfa","Type":"ContainerStarted","Data":"83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a"}
Jul 06 21:12:46.824860 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:46.824805    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podStartSLOduration=3.244161239 podStartE2EDuration="8.82477656s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.580615321s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:46.821915369 +0000 UTC m=+2264.156255985" watchObservedRunningTime="2026-07-06 21:12:46.82477656 +0000 UTC m=+2264.159117154"
Jul 06 21:12:46.915412 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:46.915374    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:12:47.008885 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:47.008844    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:49.009223 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:49.009183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:12:49.011058 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:49.011030    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:12:51.218129 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:51.218068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:12:56.915520 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:56.915463    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:12:59.008983 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:12:59.008917    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:13:06.915778 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:06.915724    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:13:09.009060 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:09.009010    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:13:16.915370 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:16.915321    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:13:19.009216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:19.009174    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:13:26.915600 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:26.915535    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:13:29.009203 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:29.009157    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:13:36.915635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:36.915590    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:13:39.009116 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:39.009071    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:13:46.916090 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:46.916044    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:13:49.009210 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:49.009162    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:13:56.916136 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:56.916027    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:13:59.009546 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:13:59.009505    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:14:06.915743 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:06.915693    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:14:09.009743 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:09.009704    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:14:16.915552 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:16.915504    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:14:19.009415 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:19.009362    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:14:26.916111 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:26.916056    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:14:29.009098 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:29.009052    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:14:36.915659 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:36.915615    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:14:39.009471 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:39.009426    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:14:46.926392 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:46.926354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:14:46.934601 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:46.934574    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:14:49.009471 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:49.009421    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:14:54.584873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:54.584832    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"]
Jul 06 21:14:54.619736 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:54.615239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:55.870884 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:55.870847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:14:59.019565 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:59.019536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:14:59.027582 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:14:59.027556    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:15:00.337099 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:00.337063    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"]
Jul 06 21:15:00.367766 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:00.367740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:15:00.663732 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:00.663633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:15:00.764372 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:00.764334    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:15:03.411257 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:03.411223    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:15:03.412771 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:03.412748    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:15:09.609828 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:09.609789    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" containerID="cri-o://cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472" gracePeriod=45
Jul 06 21:15:09.927792 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:09.927752    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 06 21:15:10.286357 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.286329    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:15:10.341900 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.341868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-dshm\") pod \"53a0f86f-4290-419a-9913-6df2287757a9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") "
Jul 06 21:15:10.342102 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.341908    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a0f86f-4290-419a-9913-6df2287757a9-tls-certs\") pod \"53a0f86f-4290-419a-9913-6df2287757a9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") "
Jul 06 21:15:10.342102 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.341942    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-model-cache\") pod \"53a0f86f-4290-419a-9913-6df2287757a9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") "
Jul 06 21:15:10.342102 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.342009    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-tmp-dir\") pod \"53a0f86f-4290-419a-9913-6df2287757a9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") "
Jul 06 21:15:10.342102 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.342073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-home\") pod \"53a0f86f-4290-419a-9913-6df2287757a9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") "
Jul 06 21:15:10.342327 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.342128    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9dnv\" (UniqueName: \"kubernetes.io/projected/53a0f86f-4290-419a-9913-6df2287757a9-kube-api-access-z9dnv\") pod \"53a0f86f-4290-419a-9913-6df2287757a9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") "
Jul 06 21:15:10.342327 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.342191    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-kserve-provision-location\") pod \"53a0f86f-4290-419a-9913-6df2287757a9\" (UID: \"53a0f86f-4290-419a-9913-6df2287757a9\") "
Jul 06 21:15:10.342327 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.342253    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "53a0f86f-4290-419a-9913-6df2287757a9" (UID: "53a0f86f-4290-419a-9913-6df2287757a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:10.342556 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.342532    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:10.342788 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.342761    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-home" (OuterVolumeSpecName: "home") pod "53a0f86f-4290-419a-9913-6df2287757a9" (UID: "53a0f86f-4290-419a-9913-6df2287757a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:10.344639 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.344611    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-dshm" (OuterVolumeSpecName: "dshm") pod "53a0f86f-4290-419a-9913-6df2287757a9" (UID: "53a0f86f-4290-419a-9913-6df2287757a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:10.344755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.344611    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53a0f86f-4290-419a-9913-6df2287757a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53a0f86f-4290-419a-9913-6df2287757a9" (UID: "53a0f86f-4290-419a-9913-6df2287757a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:15:10.344929 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.344900    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a0f86f-4290-419a-9913-6df2287757a9-kube-api-access-z9dnv" (OuterVolumeSpecName: "kube-api-access-z9dnv") pod "53a0f86f-4290-419a-9913-6df2287757a9" (UID: "53a0f86f-4290-419a-9913-6df2287757a9"). InnerVolumeSpecName "kube-api-access-z9dnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:15:10.357721 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.357688    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53a0f86f-4290-419a-9913-6df2287757a9" (UID: "53a0f86f-4290-419a-9913-6df2287757a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:10.408947 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.408889    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53a0f86f-4290-419a-9913-6df2287757a9" (UID: "53a0f86f-4290-419a-9913-6df2287757a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:10.443285 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.443251    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:10.443285 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.443281    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a0f86f-4290-419a-9913-6df2287757a9-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:10.443478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.443294    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:10.443478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.443305    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:10.443478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.443317    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9dnv\" (UniqueName: \"kubernetes.io/projected/53a0f86f-4290-419a-9913-6df2287757a9-kube-api-access-z9dnv\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:10.443478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.443328    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53a0f86f-4290-419a-9913-6df2287757a9-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:10.715384 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.715342    2582 generic.go:356] "Generic (PLEG): container finished" podID="53a0f86f-4290-419a-9913-6df2287757a9" containerID="cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472" exitCode=0
Jul 06 21:15:10.715871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.715403    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"
Jul 06 21:15:10.715871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.715435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" event={"ID":"53a0f86f-4290-419a-9913-6df2287757a9","Type":"ContainerDied","Data":"cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472"}
Jul 06 21:15:10.715871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.715497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9" event={"ID":"53a0f86f-4290-419a-9913-6df2287757a9","Type":"ContainerDied","Data":"7eb0304618d30c2e1f3853cc66ace8e7c1ef47eea56b94d4fd07ed7b703bc978"}
Jul 06 21:15:10.715871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.715520    2582 scope.go:117] "RemoveContainer" containerID="cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472"
Jul 06 21:15:10.724957 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.724941    2582 scope.go:117] "RemoveContainer" containerID="ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c"
Jul 06 21:15:10.733436 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.733415    2582 scope.go:117] "RemoveContainer" containerID="cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472"
Jul 06 21:15:10.733680 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:15:10.733663    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472\": container with ID starting with cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472 not found: ID does not exist" containerID="cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472"
Jul 06 21:15:10.733720 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.733689    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472"} err="failed to get container status \"cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472\": rpc error: code = NotFound desc = could not find container \"cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472\": container with ID starting with cc7a80fbac4492c29751a1175f87b13f5561b39bc78b93f097e6460e5bf54472 not found: ID does not exist"
Jul 06 21:15:10.733720 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.733705    2582 scope.go:117] "RemoveContainer" containerID="ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c"
Jul 06 21:15:10.733946 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:15:10.733928    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c\": container with ID starting with ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c not found: ID does not exist" containerID="ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c"
Jul 06 21:15:10.734006 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.733953    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c"} err="failed to get container status \"ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c\": rpc error: code = NotFound desc = could not find container \"ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c\": container with ID starting with ccec061da37494b5e4cde88c32d66797c3d21ae6233faae5cef1b396ae668f0c not found: ID does not exist"
Jul 06 21:15:10.741629 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.741606    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"]
Jul 06 21:15:10.748249 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:10.748225    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9"]
Jul 06 21:15:11.337347 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:11.337308    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53a0f86f-4290-419a-9913-6df2287757a9" path="/var/lib/kubelet/pods/53a0f86f-4290-419a-9913-6df2287757a9/volumes"
Jul 06 21:15:15.681849 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:15.681809    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" containerID="cri-o://83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a" gracePeriod=45
Jul 06 21:15:16.020312 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.020220    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 06 21:15:16.433665 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.433635    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:15:16.500566 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.500533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w9gs\" (UniqueName: \"kubernetes.io/projected/b556f97e-f89b-441a-8f09-8654d0088bfa-kube-api-access-8w9gs\") pod \"b556f97e-f89b-441a-8f09-8654d0088bfa\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") "
Jul 06 21:15:16.500752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.500574    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-kserve-provision-location\") pod \"b556f97e-f89b-441a-8f09-8654d0088bfa\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") "
Jul 06 21:15:16.500752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.500606    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-home\") pod \"b556f97e-f89b-441a-8f09-8654d0088bfa\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") "
Jul 06 21:15:16.500881 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.500774    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b556f97e-f89b-441a-8f09-8654d0088bfa-tls-certs\") pod \"b556f97e-f89b-441a-8f09-8654d0088bfa\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") "
Jul 06 21:15:16.500881 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.500807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-dshm\") pod \"b556f97e-f89b-441a-8f09-8654d0088bfa\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") "
Jul 06 21:15:16.500881 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.500856    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-model-cache\") pod \"b556f97e-f89b-441a-8f09-8654d0088bfa\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") "
Jul 06 21:15:16.500881 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.500878    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-tmp-dir\") pod \"b556f97e-f89b-441a-8f09-8654d0088bfa\" (UID: \"b556f97e-f89b-441a-8f09-8654d0088bfa\") "
Jul 06 21:15:16.501369 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.501343    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "b556f97e-f89b-441a-8f09-8654d0088bfa" (UID: "b556f97e-f89b-441a-8f09-8654d0088bfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:16.501476 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.501405    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-home" (OuterVolumeSpecName: "home") pod "b556f97e-f89b-441a-8f09-8654d0088bfa" (UID: "b556f97e-f89b-441a-8f09-8654d0088bfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:16.503270 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.503233    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b556f97e-f89b-441a-8f09-8654d0088bfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b556f97e-f89b-441a-8f09-8654d0088bfa" (UID: "b556f97e-f89b-441a-8f09-8654d0088bfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:15:16.503270 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.503250    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b556f97e-f89b-441a-8f09-8654d0088bfa-kube-api-access-8w9gs" (OuterVolumeSpecName: "kube-api-access-8w9gs") pod "b556f97e-f89b-441a-8f09-8654d0088bfa" (UID: "b556f97e-f89b-441a-8f09-8654d0088bfa"). InnerVolumeSpecName "kube-api-access-8w9gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:15:16.503394 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.503317    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-dshm" (OuterVolumeSpecName: "dshm") pod "b556f97e-f89b-441a-8f09-8654d0088bfa" (UID: "b556f97e-f89b-441a-8f09-8654d0088bfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:16.518833 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.518803    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b556f97e-f89b-441a-8f09-8654d0088bfa" (UID: "b556f97e-f89b-441a-8f09-8654d0088bfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:16.568086 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.567988    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b556f97e-f89b-441a-8f09-8654d0088bfa" (UID: "b556f97e-f89b-441a-8f09-8654d0088bfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:15:16.602355 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.602323    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9gs\" (UniqueName: \"kubernetes.io/projected/b556f97e-f89b-441a-8f09-8654d0088bfa-kube-api-access-8w9gs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:16.602355 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.602353    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:16.602522 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.602364    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:16.602522 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.602383    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b556f97e-f89b-441a-8f09-8654d0088bfa-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:16.602522 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.602392    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:16.602522 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.602400    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:16.602522 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.602407    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b556f97e-f89b-441a-8f09-8654d0088bfa-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:15:16.752535 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.752498    2582 generic.go:356] "Generic (PLEG): container finished" podID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerID="83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a" exitCode=0
Jul 06 21:15:16.753016 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.752545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" event={"ID":"b556f97e-f89b-441a-8f09-8654d0088bfa","Type":"ContainerDied","Data":"83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a"}
Jul 06 21:15:16.753016 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.752560    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"
Jul 06 21:15:16.753016 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.752594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45" event={"ID":"b556f97e-f89b-441a-8f09-8654d0088bfa","Type":"ContainerDied","Data":"1320b07a9af58c7d97b7339d4f6bae935162a13782bea2429fe935d7a63ad97d"}
Jul 06 21:15:16.753016 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.752619    2582 scope.go:117] "RemoveContainer" containerID="83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a"
Jul 06 21:15:16.762155 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.762136    2582 scope.go:117] "RemoveContainer" containerID="87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887"
Jul 06 21:15:16.772544 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.772498    2582 scope.go:117] "RemoveContainer" containerID="83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a"
Jul 06 21:15:16.772869 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:15:16.772847    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a\": container with ID starting with 83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a not found: ID does not exist" containerID="83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a"
Jul 06 21:15:16.773003 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.772876    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a"} err="failed to get container status \"83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a\": rpc error: code = NotFound desc = could not find container \"83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a\": container with ID starting with 83ae4c76d6e539d858940022530283caab3580ade22d138dfeef1a7b0344747a not found: ID does not exist"
Jul 06 21:15:16.773003 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.772896    2582 scope.go:117] "RemoveContainer" containerID="87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887"
Jul 06 21:15:16.773162 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:15:16.773146    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887\": container with ID starting with 87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887 not found: ID does not exist" containerID="87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887"
Jul 06 21:15:16.773217 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.773166    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887"} err="failed to get container status \"87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887\": rpc error: code = NotFound desc = could not find container \"87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887\": container with ID starting with 87d49d5c95af6ef044e00593471c3e18eaa758cc231f80e6a472ba65b55f3887 not found: ID does not exist"
Jul 06 21:15:16.778935 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.778911    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"]
Jul 06 21:15:16.783310 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:16.783285    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45"]
Jul 06 21:15:17.336642 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:17.336610    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" path="/var/lib/kubelet/pods/b556f97e-f89b-441a-8f09-8654d0088bfa/volumes"
Jul 06 21:15:46.931551 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.931517    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"]
Jul 06 21:15:46.931998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.931990    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main"
Jul 06 21:15:46.932048 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932003    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main"
Jul 06 21:15:46.932048 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932026    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="storage-initializer"
Jul 06 21:15:46.932048 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932032    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="storage-initializer"
Jul 06 21:15:46.932048 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932040    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main"
Jul 06 21:15:46.932048 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932047    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main"
Jul 06 21:15:46.932202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932060    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="storage-initializer"
Jul 06 21:15:46.932202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932067    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="storage-initializer"
Jul 06 21:15:46.932202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932144    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b556f97e-f89b-441a-8f09-8654d0088bfa" containerName="main"
Jul 06 21:15:46.932202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.932154    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53a0f86f-4290-419a-9913-6df2287757a9" containerName="main"
Jul 06 21:15:46.937679 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.937652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:46.942089 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.942060    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 06 21:15:46.942430 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.942408    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-skw4w\""
Jul 06 21:15:46.942561 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.942440    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:15:46.942625 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.942572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:15:46.952025 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.952000    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"]
Jul 06 21:15:46.961537 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.961493    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"]
Jul 06 21:15:46.965211 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.965193    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:46.981155 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:46.981129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"]
Jul 06 21:15:47.013715 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.013684    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:15:47.078351 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.078531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.078531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.078531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.078531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8642ec-3469-475a-b805-8103cc6bc667-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.078692 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krgp\" (UniqueName: \"kubernetes.io/projected/4d8642ec-3469-475a-b805-8103cc6bc667-kube-api-access-6krgp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.078692 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.078692 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.078692 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.078841 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ba4739-47ba-43ca-88a8-0ef013aa8717-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.078841 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.078841 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbp5\" (UniqueName: \"kubernetes.io/projected/90ba4739-47ba-43ca-88a8-0ef013aa8717-kube-api-access-cxbp5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.078841 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.078841 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.078808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179310 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ba4739-47ba-43ca-88a8-0ef013aa8717-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbp5\" (UniqueName: \"kubernetes.io/projected/90ba4739-47ba-43ca-88a8-0ef013aa8717-kube-api-access-cxbp5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179513 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179790    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179802    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179790    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179838    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179855    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.179998 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.179980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.180477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.180009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8642ec-3469-475a-b805-8103cc6bc667-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.180477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.180028    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.180477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.180039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krgp\" (UniqueName: \"kubernetes.io/projected/4d8642ec-3469-475a-b805-8103cc6bc667-kube-api-access-6krgp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.180477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.180222    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.180477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.180245    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.182330 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.182255    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ba4739-47ba-43ca-88a8-0ef013aa8717-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.182470 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.182347    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8642ec-3469-475a-b805-8103cc6bc667-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.182470 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.182422    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.182470 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.182457    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.191622 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.191599    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krgp\" (UniqueName: \"kubernetes.io/projected/4d8642ec-3469-475a-b805-8103cc6bc667-kube-api-access-6krgp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.192421 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.192400    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbp5\" (UniqueName: \"kubernetes.io/projected/90ba4739-47ba-43ca-88a8-0ef013aa8717-kube-api-access-cxbp5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.249405 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.249344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:47.278114 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.278046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:47.325839 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.325782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:15:47.418432 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:15:47.418400    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8642ec_3469_475a_b805_8103cc6bc667.slice/crio-75e3f5e044ab15ec954799caf3367bc0c3120fdcb5b021d8612398eeaad5980e WatchSource:0}: Error finding container 75e3f5e044ab15ec954799caf3367bc0c3120fdcb5b021d8612398eeaad5980e: Status 404 returned error can't find the container with id 75e3f5e044ab15ec954799caf3367bc0c3120fdcb5b021d8612398eeaad5980e
Jul 06 21:15:47.419148 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.419118    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"]
Jul 06 21:15:47.444612 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.444584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"]
Jul 06 21:15:47.446647 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:15:47.446619    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ba4739_47ba_43ca_88a8_0ef013aa8717.slice/crio-07f6cd25f5ea99b591b5b134deeacc16b07d701463d94898049e7115097e9f40 WatchSource:0}: Error finding container 07f6cd25f5ea99b591b5b134deeacc16b07d701463d94898049e7115097e9f40: Status 404 returned error can't find the container with id 07f6cd25f5ea99b591b5b134deeacc16b07d701463d94898049e7115097e9f40
Jul 06 21:15:47.928312 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.928279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" event={"ID":"90ba4739-47ba-43ca-88a8-0ef013aa8717","Type":"ContainerStarted","Data":"d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368"}
Jul 06 21:15:47.928501 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.928346    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" event={"ID":"90ba4739-47ba-43ca-88a8-0ef013aa8717","Type":"ContainerStarted","Data":"07f6cd25f5ea99b591b5b134deeacc16b07d701463d94898049e7115097e9f40"}
Jul 06 21:15:47.930051 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.930022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerStarted","Data":"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"}
Jul 06 21:15:47.930162 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.930079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerStarted","Data":"75e3f5e044ab15ec954799caf3367bc0c3120fdcb5b021d8612398eeaad5980e"}
Jul 06 21:15:47.930162 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:47.930114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:48.939679 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:48.939639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerStarted","Data":"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"}
Jul 06 21:15:52.972032 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:52.971997    2582 generic.go:356] "Generic (PLEG): container finished" podID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerID="d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368" exitCode=0
Jul 06 21:15:52.972427 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:52.972071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" event={"ID":"90ba4739-47ba-43ca-88a8-0ef013aa8717","Type":"ContainerDied","Data":"d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368"}
Jul 06 21:15:53.980576 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:53.980539    2582 generic.go:356] "Generic (PLEG): container finished" podID="4d8642ec-3469-475a-b805-8103cc6bc667" containerID="3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009" exitCode=0
Jul 06 21:15:53.981087 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:53.980606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerDied","Data":"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"}
Jul 06 21:15:53.983606 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:53.983578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" event={"ID":"90ba4739-47ba-43ca-88a8-0ef013aa8717","Type":"ContainerStarted","Data":"868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab"}
Jul 06 21:15:54.041164 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:54.041117    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podStartSLOduration=3.064652511 podStartE2EDuration="8.041101313s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.976448802s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:54.037174899 +0000 UTC m=+2451.371515523" watchObservedRunningTime="2026-07-06 21:15:54.041101313 +0000 UTC m=+2451.375441908"
Jul 06 21:15:54.992095 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:54.992059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerStarted","Data":"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"}
Jul 06 21:15:55.026514 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:55.026444    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podStartSLOduration=3.753285979 podStartE2EDuration="9.026420991s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.273135012s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:55.023920375 +0000 UTC m=+2452.358260991" watchObservedRunningTime="2026-07-06 21:15:55.026420991 +0000 UTC m=+2452.360761587"
Jul 06 21:15:57.249759 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:57.249705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:57.250223 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:57.249839    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:57.251453 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:57.251417    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:15:57.263534 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:57.263505    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:15:57.279118 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:57.279090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:57.279283 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:57.279158    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:15:57.280759 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:15:57.280723    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:16:07.250459 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:07.250410    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:16:07.278821 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:07.278769    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:16:10.015697 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.015653    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"]
Jul 06 21:16:10.019310 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.019290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.024564 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.024536    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 06 21:16:10.036499 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.036467    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"]
Jul 06 21:16:10.064131 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.064098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:16:10.128301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.128259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tmp-dir\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.128447 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.128325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8qs\" (UniqueName: \"kubernetes.io/projected/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kube-api-access-rr8qs\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.128505 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.128444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-home\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.128505 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.128482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-dshm\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.128575 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.128511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-model-cache\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.128575 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.128552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tls-certs\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.128690 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.128607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kserve-provision-location\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229283 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tmp-dir\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8qs\" (UniqueName: \"kubernetes.io/projected/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kube-api-access-rr8qs\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-home\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-dshm\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-model-cache\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229721 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tls-certs\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229721 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kserve-provision-location\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229721 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229701    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tmp-dir\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229860 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229757    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-model-cache\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.229860 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.229810    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-home\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.230124 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.230102    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kserve-provision-location\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.232032 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.231997    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-dshm\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.232181 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.232163    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tls-certs\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.238554 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.238523    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8qs\" (UniqueName: \"kubernetes.io/projected/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kube-api-access-rr8qs\") pod \"stop-feature-test-kserve-9b9d66cf-9snw6\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.330662 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.330554    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:10.506358 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.506330    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"]
Jul 06 21:16:10.508870 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:16:10.508825    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf15fc0_2bde_486c_ab81_c260eb51ac77.slice/crio-c1364855a889ce6c284eb3517ed55a28633264cad548eb9eba85d0652c5d82ae WatchSource:0}: Error finding container c1364855a889ce6c284eb3517ed55a28633264cad548eb9eba85d0652c5d82ae: Status 404 returned error can't find the container with id c1364855a889ce6c284eb3517ed55a28633264cad548eb9eba85d0652c5d82ae
Jul 06 21:16:10.522641 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:10.522606    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:16:11.107824 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:11.107788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" event={"ID":"cbf15fc0-2bde-486c-ab81-c260eb51ac77","Type":"ContainerStarted","Data":"82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588"}
Jul 06 21:16:11.108285 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:11.107850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" event={"ID":"cbf15fc0-2bde-486c-ab81-c260eb51ac77","Type":"ContainerStarted","Data":"c1364855a889ce6c284eb3517ed55a28633264cad548eb9eba85d0652c5d82ae"}
Jul 06 21:16:14.128518 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:14.128482    2582 generic.go:356] "Generic (PLEG): container finished" podID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerID="82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588" exitCode=0
Jul 06 21:16:14.129015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:14.128568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" event={"ID":"cbf15fc0-2bde-486c-ab81-c260eb51ac77","Type":"ContainerDied","Data":"82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588"}
Jul 06 21:16:15.137413 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:15.137370    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" event={"ID":"cbf15fc0-2bde-486c-ab81-c260eb51ac77","Type":"ContainerStarted","Data":"f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4"}
Jul 06 21:16:15.166515 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:15.166451    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podStartSLOduration=2.737586172 podStartE2EDuration="6.166428787s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.428842615s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:16:15.164750235 +0000 UTC m=+2472.499090831" watchObservedRunningTime="2026-07-06 21:16:15.166428787 +0000 UTC m=+2472.500769383"
Jul 06 21:16:15.338625 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:15.338588    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:17.249902 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:17.249845    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:16:17.278883 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:17.278820    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:16:20.331195 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:20.331144    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:16:20.333346 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:20.333313    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:16:27.249893 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:27.249830    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:16:27.278956 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:27.278908    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:16:30.331032 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:30.330979    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:16:37.250045 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:37.249996    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:16:37.279338 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:37.279291    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:16:37.669588 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:37.669500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:16:40.331069 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:40.331010    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:16:47.249897 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:47.249849    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:16:47.278510 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:47.278461    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:16:47.706420 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:47.706383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:16:50.331381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:50.331336    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:16:57.250141 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:57.249957    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:16:57.279590 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:16:57.279547    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:17:00.331055 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:00.331007    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:17:01.690022 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:01.689985    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:07.250087 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:07.250024    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:17:07.279164 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:07.279125    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:17:10.331506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:10.331458    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:17:12.426882 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:12.426823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:17.250154 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:17.250107    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:17:17.279282 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:17.279231    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:17:20.331119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:20.331072    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:17:27.250210 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:27.250166    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:17:27.279308 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:27.279267    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:17:30.331218 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:30.331175    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:17:37.250113 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:37.250056    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:17:37.278818 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:37.278771    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:17:37.325720 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:37.325682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:17:40.331142 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:40.331088    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:17:47.250831 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:47.250781    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:17:47.278583 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:47.278537    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:17:50.331180 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:50.331137    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:17:57.250604 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:57.250558    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:17:57.278777 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:17:57.278738    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:18:00.331089 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:00.331032    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:18:07.250376 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:07.250311    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:18:07.278731 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:07.278693    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:18:10.331776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:10.331729    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:18:17.250763 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:17.250707    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:18:17.279411 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:17.279366    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:18:20.331774 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:20.331725    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:18:27.250107 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:27.250053    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:18:27.278465 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:27.278427    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:18:30.331054 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:30.331007    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:18:37.249742 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:37.249693    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:18:37.279036 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:37.278988    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:18:40.331428 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:40.331383    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:18:47.250422 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:47.250376    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:18:47.279153 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:47.279107    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:18:50.331755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:50.331699    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:18:57.249824 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:57.249778    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Jul 06 21:18:57.279402 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:18:57.279359    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:19:00.331069 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:00.331017    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:19:07.259462 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:07.259429    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:19:07.277390 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:07.277363    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:19:07.290264 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:07.290230    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:19:07.298136 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:07.298113    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:19:10.331401 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:10.331363    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:19:20.343849 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:20.343785    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:19:20.352192 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:20.352169    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:19:21.685287 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:21.685248    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"]
Jul 06 21:19:21.688755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:21.688720    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"]
Jul 06 21:19:21.697446 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:21.697415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:21.865326 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:21.865287    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:23.047664 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:23.047631    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"]
Jul 06 21:19:23.078431 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:23.078386    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:23.192992 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:23.192909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:36.712807 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:36.712758    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" containerID="cri-o://868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab" gracePeriod=45
Jul 06 21:19:36.714266 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:36.714228    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main" containerID="cri-o://11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe" gracePeriod=45
Jul 06 21:19:37.291031 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.290983    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 06 21:19:37.527900 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.527864    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="llm-d-routing-sidecar" containerID="cri-o://0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b" gracePeriod=45
Jul 06 21:19:37.679292 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.679265    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:19:37.694439 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.694407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ba4739-47ba-43ca-88a8-0ef013aa8717-tls-certs\") pod \"90ba4739-47ba-43ca-88a8-0ef013aa8717\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") "
Jul 06 21:19:37.694589 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.694472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-dshm\") pod \"90ba4739-47ba-43ca-88a8-0ef013aa8717\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") "
Jul 06 21:19:37.694589 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.694514    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxbp5\" (UniqueName: \"kubernetes.io/projected/90ba4739-47ba-43ca-88a8-0ef013aa8717-kube-api-access-cxbp5\") pod \"90ba4739-47ba-43ca-88a8-0ef013aa8717\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") "
Jul 06 21:19:37.694589 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.694545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-tmp-dir\") pod \"90ba4739-47ba-43ca-88a8-0ef013aa8717\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") "
Jul 06 21:19:37.694589 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.694577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-kserve-provision-location\") pod \"90ba4739-47ba-43ca-88a8-0ef013aa8717\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") "
Jul 06 21:19:37.694812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.694622    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-home\") pod \"90ba4739-47ba-43ca-88a8-0ef013aa8717\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") "
Jul 06 21:19:37.694812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.694653    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-model-cache\") pod \"90ba4739-47ba-43ca-88a8-0ef013aa8717\" (UID: \"90ba4739-47ba-43ca-88a8-0ef013aa8717\") "
Jul 06 21:19:37.695742 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.695344    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-model-cache" (OuterVolumeSpecName: "model-cache") pod "90ba4739-47ba-43ca-88a8-0ef013aa8717" (UID: "90ba4739-47ba-43ca-88a8-0ef013aa8717"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.696514 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.696424    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-home" (OuterVolumeSpecName: "home") pod "90ba4739-47ba-43ca-88a8-0ef013aa8717" (UID: "90ba4739-47ba-43ca-88a8-0ef013aa8717"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.699009 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.698949    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-dshm" (OuterVolumeSpecName: "dshm") pod "90ba4739-47ba-43ca-88a8-0ef013aa8717" (UID: "90ba4739-47ba-43ca-88a8-0ef013aa8717"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.699139 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.699121    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ba4739-47ba-43ca-88a8-0ef013aa8717-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90ba4739-47ba-43ca-88a8-0ef013aa8717" (UID: "90ba4739-47ba-43ca-88a8-0ef013aa8717"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:19:37.702478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.702424    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ba4739-47ba-43ca-88a8-0ef013aa8717-kube-api-access-cxbp5" (OuterVolumeSpecName: "kube-api-access-cxbp5") pod "90ba4739-47ba-43ca-88a8-0ef013aa8717" (UID: "90ba4739-47ba-43ca-88a8-0ef013aa8717"). InnerVolumeSpecName "kube-api-access-cxbp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:19:37.715641 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.715605    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90ba4739-47ba-43ca-88a8-0ef013aa8717" (UID: "90ba4739-47ba-43ca-88a8-0ef013aa8717"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.763098 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.763049    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90ba4739-47ba-43ca-88a8-0ef013aa8717" (UID: "90ba4739-47ba-43ca-88a8-0ef013aa8717"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.796479 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.796456    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.796542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.796484    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxbp5\" (UniqueName: \"kubernetes.io/projected/90ba4739-47ba-43ca-88a8-0ef013aa8717-kube-api-access-cxbp5\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.796542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.796516    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.796542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.796536    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.796646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.796544    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.796646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.796553    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ba4739-47ba-43ca-88a8-0ef013aa8717-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.796646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.796561    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ba4739-47ba-43ca-88a8-0ef013aa8717-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.804812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.804778    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:19:37.897152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897122    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-dshm\") pod \"4d8642ec-3469-475a-b805-8103cc6bc667\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") "
Jul 06 21:19:37.897152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897161    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-home\") pod \"4d8642ec-3469-475a-b805-8103cc6bc667\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") "
Jul 06 21:19:37.897410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8642ec-3469-475a-b805-8103cc6bc667-tls-certs\") pod \"4d8642ec-3469-475a-b805-8103cc6bc667\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") "
Jul 06 21:19:37.897410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897210    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-model-cache\") pod \"4d8642ec-3469-475a-b805-8103cc6bc667\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") "
Jul 06 21:19:37.897410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897240    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-kserve-provision-location\") pod \"4d8642ec-3469-475a-b805-8103cc6bc667\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") "
Jul 06 21:19:37.897410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6krgp\" (UniqueName: \"kubernetes.io/projected/4d8642ec-3469-475a-b805-8103cc6bc667-kube-api-access-6krgp\") pod \"4d8642ec-3469-475a-b805-8103cc6bc667\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") "
Jul 06 21:19:37.897410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897333    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-tmp-dir\") pod \"4d8642ec-3469-475a-b805-8103cc6bc667\" (UID: \"4d8642ec-3469-475a-b805-8103cc6bc667\") "
Jul 06 21:19:37.897677 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897477    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d8642ec-3469-475a-b805-8103cc6bc667" (UID: "4d8642ec-3469-475a-b805-8103cc6bc667"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.897677 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897670    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.898026 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.897999    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-home" (OuterVolumeSpecName: "home") pod "4d8642ec-3469-475a-b805-8103cc6bc667" (UID: "4d8642ec-3469-475a-b805-8103cc6bc667"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.899474 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.899449    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8642ec-3469-475a-b805-8103cc6bc667-kube-api-access-6krgp" (OuterVolumeSpecName: "kube-api-access-6krgp") pod "4d8642ec-3469-475a-b805-8103cc6bc667" (UID: "4d8642ec-3469-475a-b805-8103cc6bc667"). InnerVolumeSpecName "kube-api-access-6krgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:19:37.899809 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.899782    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-dshm" (OuterVolumeSpecName: "dshm") pod "4d8642ec-3469-475a-b805-8103cc6bc667" (UID: "4d8642ec-3469-475a-b805-8103cc6bc667"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.899956 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.899856    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8642ec-3469-475a-b805-8103cc6bc667-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d8642ec-3469-475a-b805-8103cc6bc667" (UID: "4d8642ec-3469-475a-b805-8103cc6bc667"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:19:37.910626 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.910599    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d8642ec-3469-475a-b805-8103cc6bc667" (UID: "4d8642ec-3469-475a-b805-8103cc6bc667"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.926410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.926363    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d8642ec-3469-475a-b805-8103cc6bc667" (UID: "4d8642ec-3469-475a-b805-8103cc6bc667"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:37.999151 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.999110    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krgp\" (UniqueName: \"kubernetes.io/projected/4d8642ec-3469-475a-b805-8103cc6bc667-kube-api-access-6krgp\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.999151 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.999147    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.999151 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.999161    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.999421 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.999173    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.999421 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.999187    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8642ec-3469-475a-b805-8103cc6bc667-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:37.999421 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:37.999200    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8642ec-3469-475a-b805-8103cc6bc667-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:38.072347 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.072251    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" containerID="cri-o://f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4" gracePeriod=45
Jul 06 21:19:38.345776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.345675    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 06 21:19:38.508809 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.508770    2582 generic.go:356] "Generic (PLEG): container finished" podID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerID="868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab" exitCode=0
Jul 06 21:19:38.509005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.508834    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"
Jul 06 21:19:38.509005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.508859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" event={"ID":"90ba4739-47ba-43ca-88a8-0ef013aa8717","Type":"ContainerDied","Data":"868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab"}
Jul 06 21:19:38.509005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.508922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4" event={"ID":"90ba4739-47ba-43ca-88a8-0ef013aa8717","Type":"ContainerDied","Data":"07f6cd25f5ea99b591b5b134deeacc16b07d701463d94898049e7115097e9f40"}
Jul 06 21:19:38.509005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.508948    2582 scope.go:117] "RemoveContainer" containerID="868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab"
Jul 06 21:19:38.511110 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.511088    2582 generic.go:356] "Generic (PLEG): container finished" podID="4d8642ec-3469-475a-b805-8103cc6bc667" containerID="11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe" exitCode=0
Jul 06 21:19:38.511110 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.511107    2582 generic.go:356] "Generic (PLEG): container finished" podID="4d8642ec-3469-475a-b805-8103cc6bc667" containerID="0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b" exitCode=0
Jul 06 21:19:38.511303 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.511166    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"
Jul 06 21:19:38.511303 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.511176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerDied","Data":"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"}
Jul 06 21:19:38.511303 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.511236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerDied","Data":"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"}
Jul 06 21:19:38.511303 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.511261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp" event={"ID":"4d8642ec-3469-475a-b805-8103cc6bc667","Type":"ContainerDied","Data":"75e3f5e044ab15ec954799caf3367bc0c3120fdcb5b021d8612398eeaad5980e"}
Jul 06 21:19:38.520873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.520851    2582 scope.go:117] "RemoveContainer" containerID="d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368"
Jul 06 21:19:38.530444 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.530424    2582 scope.go:117] "RemoveContainer" containerID="868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab"
Jul 06 21:19:38.530729 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:19:38.530711    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab\": container with ID starting with 868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab not found: ID does not exist" containerID="868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab"
Jul 06 21:19:38.530781 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.530739    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab"} err="failed to get container status \"868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab\": rpc error: code = NotFound desc = could not find container \"868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab\": container with ID starting with 868f916dd211102229cbdb9586f84626a29b03af8109c1f09d81644dab4418ab not found: ID does not exist"
Jul 06 21:19:38.530781 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.530756    2582 scope.go:117] "RemoveContainer" containerID="d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368"
Jul 06 21:19:38.531010 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:19:38.530992    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368\": container with ID starting with d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368 not found: ID does not exist" containerID="d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368"
Jul 06 21:19:38.531096 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.531020    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368"} err="failed to get container status \"d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368\": rpc error: code = NotFound desc = could not find container \"d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368\": container with ID starting with d7a6b0aca7faab0a936a6e09707ac02e96bf5354b7ad6b4cdbfde00e2947b368 not found: ID does not exist"
Jul 06 21:19:38.531096 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.531041    2582 scope.go:117] "RemoveContainer" containerID="11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"
Jul 06 21:19:38.535315 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.535294    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"]
Jul 06 21:19:38.540143 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.539950    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4"]
Jul 06 21:19:38.540780 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.540765    2582 scope.go:117] "RemoveContainer" containerID="3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"
Jul 06 21:19:38.549173 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.549150    2582 scope.go:117] "RemoveContainer" containerID="0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"
Jul 06 21:19:38.552783 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.552762    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"]
Jul 06 21:19:38.558491 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.558446    2582 scope.go:117] "RemoveContainer" containerID="11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"
Jul 06 21:19:38.558804 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:19:38.558745    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe\": container with ID starting with 11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe not found: ID does not exist" containerID="11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"
Jul 06 21:19:38.558894 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.558798    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"} err="failed to get container status \"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe\": rpc error: code = NotFound desc = could not find container \"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe\": container with ID starting with 11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe not found: ID does not exist"
Jul 06 21:19:38.558894 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.558824    2582 scope.go:117] "RemoveContainer" containerID="3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"
Jul 06 21:19:38.559130 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:19:38.559107    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009\": container with ID starting with 3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009 not found: ID does not exist" containerID="3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"
Jul 06 21:19:38.559193 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.559138    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"} err="failed to get container status \"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009\": rpc error: code = NotFound desc = could not find container \"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009\": container with ID starting with 3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009 not found: ID does not exist"
Jul 06 21:19:38.559193 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.559161    2582 scope.go:117] "RemoveContainer" containerID="0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"
Jul 06 21:19:38.559478 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:19:38.559451    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b\": container with ID starting with 0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b not found: ID does not exist" containerID="0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"
Jul 06 21:19:38.559587 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.559491    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"} err="failed to get container status \"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b\": rpc error: code = NotFound desc = could not find container \"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b\": container with ID starting with 0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b not found: ID does not exist"
Jul 06 21:19:38.559587 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.559515    2582 scope.go:117] "RemoveContainer" containerID="11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"
Jul 06 21:19:38.559828 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.559796    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe"} err="failed to get container status \"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe\": rpc error: code = NotFound desc = could not find container \"11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe\": container with ID starting with 11689a82d4c537ddce9caa1b7fd76008468c3cf2c45ac7a36ccb260747d35ebe not found: ID does not exist"
Jul 06 21:19:38.559888 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.559826    2582 scope.go:117] "RemoveContainer" containerID="3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"
Jul 06 21:19:38.560129 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.560105    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009"} err="failed to get container status \"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009\": rpc error: code = NotFound desc = could not find container \"3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009\": container with ID starting with 3a8459e60cf51a6f6c05f7a3da109b421a38117188d8dc0ecae28b19bdcfc009 not found: ID does not exist"
Jul 06 21:19:38.560129 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.560132    2582 scope.go:117] "RemoveContainer" containerID="0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"
Jul 06 21:19:38.560274 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.560197    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp"]
Jul 06 21:19:38.560380 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.560357    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b"} err="failed to get container status \"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b\": rpc error: code = NotFound desc = could not find container \"0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b\": container with ID starting with 0d134fbbd13d1c74030cb34d743d970f92d0f3efffe06117d06c0865448e218b not found: ID does not exist"
Jul 06 21:19:38.837631 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.837608    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:19:38.908862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.908778    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-dshm\") pod \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") "
Jul 06 21:19:38.908862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.908811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tls-certs\") pod \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") "
Jul 06 21:19:38.908862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.908832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tmp-dir\") pod \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") "
Jul 06 21:19:38.909167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.908882    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kserve-provision-location\") pod \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") "
Jul 06 21:19:38.909167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.908921    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr8qs\" (UniqueName: \"kubernetes.io/projected/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kube-api-access-rr8qs\") pod \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") "
Jul 06 21:19:38.909167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.908998    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-model-cache\") pod \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") "
Jul 06 21:19:38.909167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.909062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-home\") pod \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\" (UID: \"cbf15fc0-2bde-486c-ab81-c260eb51ac77\") "
Jul 06 21:19:38.909777 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.909751    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-home" (OuterVolumeSpecName: "home") pod "cbf15fc0-2bde-486c-ab81-c260eb51ac77" (UID: "cbf15fc0-2bde-486c-ab81-c260eb51ac77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:38.910143 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.910116    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbf15fc0-2bde-486c-ab81-c260eb51ac77" (UID: "cbf15fc0-2bde-486c-ab81-c260eb51ac77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:38.911106 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.911080    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbf15fc0-2bde-486c-ab81-c260eb51ac77" (UID: "cbf15fc0-2bde-486c-ab81-c260eb51ac77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:19:38.911456 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.911433    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kube-api-access-rr8qs" (OuterVolumeSpecName: "kube-api-access-rr8qs") pod "cbf15fc0-2bde-486c-ab81-c260eb51ac77" (UID: "cbf15fc0-2bde-486c-ab81-c260eb51ac77"). InnerVolumeSpecName "kube-api-access-rr8qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:19:38.911456 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.911440    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-dshm" (OuterVolumeSpecName: "dshm") pod "cbf15fc0-2bde-486c-ab81-c260eb51ac77" (UID: "cbf15fc0-2bde-486c-ab81-c260eb51ac77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:38.923955 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.923934    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbf15fc0-2bde-486c-ab81-c260eb51ac77" (UID: "cbf15fc0-2bde-486c-ab81-c260eb51ac77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:38.933425 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:38.933384    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbf15fc0-2bde-486c-ab81-c260eb51ac77" (UID: "cbf15fc0-2bde-486c-ab81-c260eb51ac77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:19:39.010338 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.010303    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:39.010338 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.010332    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:39.010338 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.010342    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:39.010571 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.010350    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:39.010571 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.010367    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr8qs\" (UniqueName: \"kubernetes.io/projected/cbf15fc0-2bde-486c-ab81-c260eb51ac77-kube-api-access-rr8qs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:39.010571 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.010376    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:39.010571 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.010384    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf15fc0-2bde-486c-ab81-c260eb51ac77-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:19:39.337464 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.337427    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" path="/var/lib/kubelet/pods/4d8642ec-3469-475a-b805-8103cc6bc667/volumes"
Jul 06 21:19:39.338036 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.338016    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" path="/var/lib/kubelet/pods/90ba4739-47ba-43ca-88a8-0ef013aa8717/volumes"
Jul 06 21:19:39.520730 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.520696    2582 generic.go:356] "Generic (PLEG): container finished" podID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerID="f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4" exitCode=0
Jul 06 21:19:39.520947 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.520758    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"
Jul 06 21:19:39.520947 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.520763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" event={"ID":"cbf15fc0-2bde-486c-ab81-c260eb51ac77","Type":"ContainerDied","Data":"f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4"}
Jul 06 21:19:39.520947 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.520879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6" event={"ID":"cbf15fc0-2bde-486c-ab81-c260eb51ac77","Type":"ContainerDied","Data":"c1364855a889ce6c284eb3517ed55a28633264cad548eb9eba85d0652c5d82ae"}
Jul 06 21:19:39.520947 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.520903    2582 scope.go:117] "RemoveContainer" containerID="f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4"
Jul 06 21:19:39.529677 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.529660    2582 scope.go:117] "RemoveContainer" containerID="82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588"
Jul 06 21:19:39.537848 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.537828    2582 scope.go:117] "RemoveContainer" containerID="f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4"
Jul 06 21:19:39.538106 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:19:39.538087    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4\": container with ID starting with f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4 not found: ID does not exist" containerID="f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4"
Jul 06 21:19:39.538191 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.538116    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4"} err="failed to get container status \"f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4\": rpc error: code = NotFound desc = could not find container \"f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4\": container with ID starting with f44add8474cea3511a73ace0bcec3baba1019d9b2b71ead680e0a11a3db4feb4 not found: ID does not exist"
Jul 06 21:19:39.538191 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.538139    2582 scope.go:117] "RemoveContainer" containerID="82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588"
Jul 06 21:19:39.538389 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:19:39.538370    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588\": container with ID starting with 82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588 not found: ID does not exist" containerID="82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588"
Jul 06 21:19:39.538430 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.538396    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588"} err="failed to get container status \"82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588\": rpc error: code = NotFound desc = could not find container \"82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588\": container with ID starting with 82337cdcf72cbbbe54c46e93b175539aec5257fb9ed80fa08f26f88cca99b588 not found: ID does not exist"
Jul 06 21:19:39.547777 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.547755    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"]
Jul 06 21:19:39.554732 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:39.554707    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6"]
Jul 06 21:19:41.342608 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:41.342574    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" path="/var/lib/kubelet/pods/cbf15fc0-2bde-486c-ab81-c260eb51ac77/volumes"
Jul 06 21:19:50.664546 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.664512    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"]
Jul 06 21:19:50.664953 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.664938    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="storage-initializer"
Jul 06 21:19:50.664953 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.664950    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="storage-initializer"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.664974    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.664982    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.664994    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="llm-d-routing-sidecar"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.664999    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="llm-d-routing-sidecar"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665009    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665026    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665031    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665037    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="storage-initializer"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665042    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="storage-initializer"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665048    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="storage-initializer"
Jul 06 21:19:50.665063 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665053    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="storage-initializer"
Jul 06 21:19:50.665443 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665126    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ba4739-47ba-43ca-88a8-0ef013aa8717" containerName="main"
Jul 06 21:19:50.665443 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665135    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="main"
Jul 06 21:19:50.665443 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665147    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8642ec-3469-475a-b805-8103cc6bc667" containerName="llm-d-routing-sidecar"
Jul 06 21:19:50.665443 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.665152    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf15fc0-2bde-486c-ab81-c260eb51ac77" containerName="main"
Jul 06 21:19:50.668526 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.668510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.672415 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.672374    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:19:50.672624 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.672611    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 21:19:50.673376 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.673357    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:19:50.673496 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.673382    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 06 21:19:50.686480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.686455    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"]
Jul 06 21:19:50.723480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tmp-dir\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.723480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-home\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.723480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5af5fefd-2f75-44ea-9a08-4424e3c06b67-cabundle-cert\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.723480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-model-cache\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.723480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tls-certs\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.723480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.723825 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dsv\" (UniqueName: \"kubernetes.io/projected/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kube-api-access-m8dsv\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.723825 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.723669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-dshm\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.779500 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.779463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:50.824975 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.824919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5af5fefd-2f75-44ea-9a08-4424e3c06b67-cabundle-cert\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.824991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-model-cache\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tls-certs\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dsv\" (UniqueName: \"kubernetes.io/projected/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kube-api-access-m8dsv\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-dshm\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tmp-dir\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-home\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-model-cache\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825536    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825657 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825617    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-home\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825705 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825653    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tmp-dir\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.825747 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.825716    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5af5fefd-2f75-44ea-9a08-4424e3c06b67-cabundle-cert\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.827663 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.827637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-dshm\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.827771 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.827705    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tls-certs\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.836346 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.836321    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dsv\" (UniqueName: \"kubernetes.io/projected/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kube-api-access-m8dsv\") pod \"lora-single-adapter-test-kserve-58678c44f4-sbltp\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:50.978796 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:50.978759    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:51.137953 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:51.137927    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"]
Jul 06 21:19:51.140368 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:19:51.140333    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af5fefd_2f75_44ea_9a08_4424e3c06b67.slice/crio-c71ae939a1e3d06d77a45080e7ac92453de5a90bb1d2bf72420878db637f1389 WatchSource:0}: Error finding container c71ae939a1e3d06d77a45080e7ac92453de5a90bb1d2bf72420878db637f1389: Status 404 returned error can't find the container with id c71ae939a1e3d06d77a45080e7ac92453de5a90bb1d2bf72420878db637f1389
Jul 06 21:19:51.591403 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:51.591370    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" event={"ID":"5af5fefd-2f75-44ea-9a08-4424e3c06b67","Type":"ContainerStarted","Data":"75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9"}
Jul 06 21:19:51.591681 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:51.591428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" event={"ID":"5af5fefd-2f75-44ea-9a08-4424e3c06b67","Type":"ContainerStarted","Data":"c71ae939a1e3d06d77a45080e7ac92453de5a90bb1d2bf72420878db637f1389"}
Jul 06 21:19:55.618187 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:55.618152    2582 generic.go:356] "Generic (PLEG): container finished" podID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerID="75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9" exitCode=0
Jul 06 21:19:55.618600 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:55.618220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" event={"ID":"5af5fefd-2f75-44ea-9a08-4424e3c06b67","Type":"ContainerDied","Data":"75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9"}
Jul 06 21:19:56.626159 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:56.626124    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" event={"ID":"5af5fefd-2f75-44ea-9a08-4424e3c06b67","Type":"ContainerStarted","Data":"1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9"}
Jul 06 21:19:56.658341 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:56.658284    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podStartSLOduration=3.156735028 podStartE2EDuration="6.658266536s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.501531508s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:19:56.654619061 +0000 UTC m=+2693.988959683" watchObservedRunningTime="2026-07-06 21:19:56.658266536 +0000 UTC m=+2693.992607130"
Jul 06 21:19:56.979420 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:56.979384    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:19:59.219397 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.219354    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"]
Jul 06 21:19:59.225096 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.225071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.228250 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.228170    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 06 21:19:59.228485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.228466    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-s2wlm\""
Jul 06 21:19:59.248943 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.248913    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"]
Jul 06 21:19:59.252476 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.252439    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"]
Jul 06 21:19:59.260059 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.260019    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.264402 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.264373    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"]
Jul 06 21:19:59.293345 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.293311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:19:59.314679 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.314638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.314907 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.314796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.314907 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.314861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-dshm\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.315076 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.314917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.315076 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.314982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.315076 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.315024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmbf\" (UniqueName: \"kubernetes.io/projected/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kube-api-access-vjmbf\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.315076 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.315066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-home\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.416394 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.416591 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.416591 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.416591 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.416591 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmbf\" (UniqueName: \"kubernetes.io/projected/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kube-api-access-vjmbf\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.416823 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.416823 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-home\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.416823 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.416823 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.416823 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416749    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.416823 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.417240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416821    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.417240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.417240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e3fa00-b567-4615-ab78-00027c8a901f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.417240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-dshm\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.417240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.416952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7w8k\" (UniqueName: \"kubernetes.io/projected/83e3fa00-b567-4615-ab78-00027c8a901f-kube-api-access-q7w8k\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.417240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.417069    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-home\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.417240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.417114    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.419303 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.419278    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-dshm\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.419648 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.419629    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.429778 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.429752    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmbf\" (UniqueName: \"kubernetes.io/projected/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kube-api-access-vjmbf\") pod \"custom-route-timeout-pd-test-kserve-76b459f96b-75c2l\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.517702 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.517613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.517702 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.517659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.517917 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.517706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.517917 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.517871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.518055 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.517950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e3fa00-b567-4615-ab78-00027c8a901f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.518055 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.518040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7w8k\" (UniqueName: \"kubernetes.io/projected/83e3fa00-b567-4615-ab78-00027c8a901f-kube-api-access-q7w8k\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.518167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.518105    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.518167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.518134    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.518167 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.518161    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.518314 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.518241    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.518453 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.518424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.520542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.520512    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.520634 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.520598    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e3fa00-b567-4615-ab78-00027c8a901f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.531365 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.531337    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7w8k\" (UniqueName: \"kubernetes.io/projected/83e3fa00-b567-4615-ab78-00027c8a901f-kube-api-access-q7w8k\") pod \"custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.552498 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.552468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:19:59.575633 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.575604    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:19:59.784408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.784380    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"]
Jul 06 21:19:59.787279 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:19:59.787244    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcd0539_2d8d_4321_b366_da5555bf3fe1.slice/crio-7faa85665d8c7166b48f772bfe1a2a1f973f193258de9d8edc9483757eb6ebe0 WatchSource:0}: Error finding container 7faa85665d8c7166b48f772bfe1a2a1f973f193258de9d8edc9483757eb6ebe0: Status 404 returned error can't find the container with id 7faa85665d8c7166b48f772bfe1a2a1f973f193258de9d8edc9483757eb6ebe0
Jul 06 21:19:59.804992 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:19:59.803229    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"]
Jul 06 21:19:59.807426 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:19:59.807392    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e3fa00_b567_4615_ab78_00027c8a901f.slice/crio-45bfc66c7c3682f855da3f34bddaba2fc66792712095a747483fd96d9f73e1f5 WatchSource:0}: Error finding container 45bfc66c7c3682f855da3f34bddaba2fc66792712095a747483fd96d9f73e1f5: Status 404 returned error can't find the container with id 45bfc66c7c3682f855da3f34bddaba2fc66792712095a747483fd96d9f73e1f5
Jul 06 21:20:00.548714 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.548681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:20:00.658388 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.658343    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" event={"ID":"83e3fa00-b567-4615-ab78-00027c8a901f","Type":"ContainerStarted","Data":"5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b"}
Jul 06 21:20:00.658588 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.658412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" event={"ID":"83e3fa00-b567-4615-ab78-00027c8a901f","Type":"ContainerStarted","Data":"45bfc66c7c3682f855da3f34bddaba2fc66792712095a747483fd96d9f73e1f5"}
Jul 06 21:20:00.660146 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.660116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerStarted","Data":"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"}
Jul 06 21:20:00.660285 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.660175    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:20:00.660285 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.660200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerStarted","Data":"7faa85665d8c7166b48f772bfe1a2a1f973f193258de9d8edc9483757eb6ebe0"}
Jul 06 21:20:00.979076 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.979040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:20:00.980653 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:00.980613    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:20:01.668394 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:01.668353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerStarted","Data":"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"}
Jul 06 21:20:03.467560 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:03.467526    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:20:03.469540 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:03.469502    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:20:04.694825 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:04.694785    2582 generic.go:356] "Generic (PLEG): container finished" podID="83e3fa00-b567-4615-ab78-00027c8a901f" containerID="5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b" exitCode=0
Jul 06 21:20:04.695307 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:04.694853    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" event={"ID":"83e3fa00-b567-4615-ab78-00027c8a901f","Type":"ContainerDied","Data":"5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b"}
Jul 06 21:20:05.703238 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:05.703202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" event={"ID":"83e3fa00-b567-4615-ab78-00027c8a901f","Type":"ContainerStarted","Data":"2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8"}
Jul 06 21:20:05.705305 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:05.705279    2582 generic.go:356] "Generic (PLEG): container finished" podID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerID="a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f" exitCode=0
Jul 06 21:20:05.705459 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:05.705353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerDied","Data":"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"}
Jul 06 21:20:05.739652 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:05.738322    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podStartSLOduration=2.214262351 podStartE2EDuration="6.738302045s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.524039694s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:20:05.732584441 +0000 UTC m=+2703.066925062" watchObservedRunningTime="2026-07-06 21:20:05.738302045 +0000 UTC m=+2703.072642642"
Jul 06 21:20:06.716221 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:06.716182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerStarted","Data":"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"}
Jul 06 21:20:06.745899 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:06.745828    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podStartSLOduration=3.7766422779999997 podStartE2EDuration="7.745805062s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.969162784s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:20:06.742671099 +0000 UTC m=+2704.077011725" watchObservedRunningTime="2026-07-06 21:20:06.745805062 +0000 UTC m=+2704.080145661"
Jul 06 21:20:09.555242 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:09.555200    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:20:09.555756 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:09.555357    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:20:09.556181 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:09.556158    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:20:09.574697 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:09.574668    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:20:09.576519 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:09.576493    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:20:09.576682 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:09.576547    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:20:09.578320 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:09.578285    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:20:10.980015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:10.979940    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:20:18.815710 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:18.815671    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="60e96a4c0c77e7a20a8252f135790aaadaf49e8b997d1902594ce4aa12b7894f" exitCode=137
Jul 06 21:20:18.816222 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:18.815730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"60e96a4c0c77e7a20a8252f135790aaadaf49e8b997d1902594ce4aa12b7894f"}
Jul 06 21:20:18.816222 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:18.815791    2582 scope.go:117] "RemoveContainer" containerID="4938f8dd2c5a03951b4e87533f49e6e7a6ac7b34ed4ca1010e3b884860664e33"
Jul 06 21:20:18.816336 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:18.816247    2582 scope.go:117] "RemoveContainer" containerID="60e96a4c0c77e7a20a8252f135790aaadaf49e8b997d1902594ce4aa12b7894f"
Jul 06 21:20:18.817577 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:18.817539    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:18.817702 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:18.817640    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:18.817702 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:18.817688    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:19.553713 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:19.553662    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:20:19.577823 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:19.577781    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:20:19.828568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:19.828460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"f6571f24946e64b1425957f047a69fb7045d88864f7b99aefe2fb0137559bbf2"}
Jul 06 21:20:20.556237 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:20.556201    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:20:20.556711 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:20.556678    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:20:20.556859 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:20.556739    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:20:20.980296 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:20.980246    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:20:21.556723 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:21.556679    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:20:21.556927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:21.556749    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:20:22.556002 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:22.555941    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:20:22.556391 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:22.556028    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:20:23.555914 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:23.555880    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:20:23.556133 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:23.555941    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:20:23.751018 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:23.750957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:20:24.556665 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:24.556622    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:20:24.557209 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:24.556678    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:20:26.557879 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:26.557833    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 06 21:20:26.558415 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:26.557932    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:20:27.104427 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:27.104391    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:20:27.105109 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:27.105076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:20:27.106830 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:27.106801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:20:29.553978 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:29.553921    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:20:29.576195 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:29.576157    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:20:30.979427 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:30.979375    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:20:39.553894 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:39.553847    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:20:39.576560 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:39.576516    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:20:40.979397 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:40.979342    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:20:49.075779 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:49.075741    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="f6571f24946e64b1425957f047a69fb7045d88864f7b99aefe2fb0137559bbf2" exitCode=137
Jul 06 21:20:49.076338 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:49.075794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"f6571f24946e64b1425957f047a69fb7045d88864f7b99aefe2fb0137559bbf2"}
Jul 06 21:20:49.076338 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:49.075874    2582 scope.go:117] "RemoveContainer" containerID="60e96a4c0c77e7a20a8252f135790aaadaf49e8b997d1902594ce4aa12b7894f"
Jul 06 21:20:49.076446 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:49.076436    2582 scope.go:117] "RemoveContainer" containerID="f6571f24946e64b1425957f047a69fb7045d88864f7b99aefe2fb0137559bbf2"
Jul 06 21:20:49.076730 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:20:49.076695    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:20:49.553632 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:49.553543    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:20:49.576839 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:49.576793    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:20:50.980157 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:50.980110    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:20:57.105261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:57.105216    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:20:57.105893 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:57.105876    2582 scope.go:117] "RemoveContainer" containerID="f6571f24946e64b1425957f047a69fb7045d88864f7b99aefe2fb0137559bbf2"
Jul 06 21:20:57.106272 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:20:57.106235    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:20:59.553789 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:59.553463    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:20:59.576454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:20:59.576406    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:21:00.979247 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:00.979204    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:21:09.331598 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:09.331565    2582 scope.go:117] "RemoveContainer" containerID="f6571f24946e64b1425957f047a69fb7045d88864f7b99aefe2fb0137559bbf2"
Jul 06 21:21:09.333726 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:09.333652    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:21:09.333820 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:09.333789    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:21:09.333875 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:09.333840    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:21:09.553985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:09.553920    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:21:09.576734 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:09.576692    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:21:10.245584 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:10.245541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"ea9d301a181b03866c20c69cf8babf07ff16a11f30943629a6027470194f9b23"}
Jul 06 21:21:10.556289 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:10.556177    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:21:10.557808 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:10.557769    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:21:10.557983 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:10.557877    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:21:10.979675 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:10.979624    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:21:11.556691 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:11.556647    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:21:11.557193 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:11.556722    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:21:12.105056 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:12.105008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:21:12.556350 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:12.556312    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:21:12.556578 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:12.556388    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:21:13.555865 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:13.555814    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:21:13.556379 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:13.555872    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:21:14.556337 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:14.556296    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:21:14.556737 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:14.556369    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:21:15.556817 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:15.556760    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:21:15.557336 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:15.556838    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:21:17.557241 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:17.557193    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 06 21:21:17.557871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:17.557821    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:21:18.134565 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:18.134516    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:21:18.140713 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:18.140679    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:21:19.553819 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:19.553774    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:21:19.576431 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:19.576388    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:21:20.979997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:20.979929    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 06 21:21:29.553680 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:29.553561    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:21:29.577003 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:29.576935    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:21:30.997242 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:30.997201    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:21:31.010472 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:31.010440    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:21:37.460287 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:37.460241    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="ea9d301a181b03866c20c69cf8babf07ff16a11f30943629a6027470194f9b23" exitCode=137
Jul 06 21:21:37.460821 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:37.460307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"ea9d301a181b03866c20c69cf8babf07ff16a11f30943629a6027470194f9b23"}
Jul 06 21:21:37.460821 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:37.460368    2582 scope.go:117] "RemoveContainer" containerID="f6571f24946e64b1425957f047a69fb7045d88864f7b99aefe2fb0137559bbf2"
Jul 06 21:21:37.460940 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:37.460825    2582 scope.go:117] "RemoveContainer" containerID="ea9d301a181b03866c20c69cf8babf07ff16a11f30943629a6027470194f9b23"
Jul 06 21:21:37.461086 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:21:37.461065    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:21:39.553088 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:39.553029    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:21:39.576067 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:39.576028    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:21:48.135638 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:48.135606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:21:48.136187 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:48.136169    2582 scope.go:117] "RemoveContainer" containerID="ea9d301a181b03866c20c69cf8babf07ff16a11f30943629a6027470194f9b23"
Jul 06 21:21:48.136428 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:21:48.136404    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:21:49.552872 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:49.552832    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:21:49.576830 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:49.576787    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:21:59.553556 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:59.553508    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:21:59.576550 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:21:59.576505    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:22:00.331842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:00.331809    2582 scope.go:117] "RemoveContainer" containerID="ea9d301a181b03866c20c69cf8babf07ff16a11f30943629a6027470194f9b23"
Jul 06 21:22:00.333084 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:00.333049    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:22:00.333219 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:00.333143    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:22:00.333219 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:00.333196    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:22:00.604211 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:00.604093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1"}
Jul 06 21:22:01.556129 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:01.556095    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:22:01.557471 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:01.557444    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:22:01.557617 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:01.557509    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:22:02.556728 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:02.556684    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:22:02.557164 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:02.556745    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:22:03.136022 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:03.135990    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:22:03.555818 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:03.555785    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:22:03.556018 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:03.555854    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:22:04.556872 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:04.556832    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:22:04.557358 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:04.556898    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:22:05.555947 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:05.555908    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:22:05.556151 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:05.555982    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:22:07.489953 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:07.489916    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:22:07.495032 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:07.495001    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:22:09.552911 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:09.552868    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:22:09.576653 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:09.576615    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:22:19.553426 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:19.553381    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:22:19.576763 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:19.576723    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:22:26.781944 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:26.781910    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1" exitCode=137
Jul 06 21:22:26.782453 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:26.781957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1"}
Jul 06 21:22:26.782453 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:26.782034    2582 scope.go:117] "RemoveContainer" containerID="ea9d301a181b03866c20c69cf8babf07ff16a11f30943629a6027470194f9b23"
Jul 06 21:22:26.782556 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:26.782482    2582 scope.go:117] "RemoveContainer" containerID="6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1"
Jul 06 21:22:26.782719 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:22:26.782691    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:22:29.553155 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:29.553107    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:22:29.576646 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:29.576610    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:22:37.491409 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:37.491372    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:22:37.491978 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:37.491943    2582 scope.go:117] "RemoveContainer" containerID="6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1"
Jul 06 21:22:37.492216 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:22:37.492192    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:22:39.553350 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:39.553305    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:22:39.576051 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:39.576004    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:22:49.552893 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:49.552849    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:22:49.576285 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:49.576248    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:22:52.331121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:52.331083    2582 scope.go:117] "RemoveContainer" containerID="6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1"
Jul 06 21:22:52.331527 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:22:52.331367    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:22:59.553457 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:59.553344    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:22:59.576029 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:22:59.575991    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:23:07.337942 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:07.337913    2582 scope.go:117] "RemoveContainer" containerID="6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1"
Jul 06 21:23:07.339040 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:07.339004    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:23:07.339121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:07.339102    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:23:07.339173 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:07.339156    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:23:08.041803 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:08.041771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"}
Jul 06 21:23:08.556386 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:08.556347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:23:08.557536 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:08.557506    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:23:08.557658 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:08.557574    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:23:09.553885 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:09.553840    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:23:09.556245 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:09.556214    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:23:09.556408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:09.556267    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:23:09.576454 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:09.576411    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:23:10.556788 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:10.556745    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:23:10.557279 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:10.556829    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:23:11.556548 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:11.556512    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:23:11.556719 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:11.556576    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:23:12.556209 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:12.556172    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:23:12.556604 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:12.556256    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:23:13.556752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:13.556718    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:23:13.557179 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:13.556774    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:23:15.527692 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:15.527653    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:23:15.528429 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:15.528254    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:23:15.529296 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:15.529280    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:23:19.553176 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:19.553115    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Jul 06 21:23:19.576610 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:19.576572    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 06 21:23:29.563120 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:29.563086    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:23:29.578159 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:29.578131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:23:29.586622 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:29.586586    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:23:29.595141 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:29.595100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:23:34.233267 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:34.233228    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e" exitCode=137
Jul 06 21:23:34.233812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:34.233292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"}
Jul 06 21:23:34.233812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:34.233354    2582 scope.go:117] "RemoveContainer" containerID="6f6ee4162174af7ff724d7ca7f2370531ae4fa618844193d63de35a67a5dcbc1"
Jul 06 21:23:34.233812 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:34.233800    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:23:34.234076 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:23:34.234054    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:23:40.031371 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:40.031339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:45.528570 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:45.528533    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:23:45.529054 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:45.528948    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:23:45.529172 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:23:45.529149    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:23:55.040774 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:55.040723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:23:58.331535 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:23:58.331508    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:23:58.331928 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:23:58.331711    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:24:12.331541 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:24:12.331481    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:24:12.332088 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:24:12.331763    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:24:26.331205 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:24:26.331127    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:24:26.331650 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:24:26.331299    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:24:38.331137 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:24:38.331101    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:24:38.331533 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:24:38.331310    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:24:51.331854 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:24:51.331823    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:24:51.332305 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:24:51.332024    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:25:03.525794 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:03.525758    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:25:03.531250 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:03.531227    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:25:05.330688 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:05.330655    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:25:05.331908 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:05.331879    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:25:05.332038 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:05.331950    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:25:05.332038 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:05.332031    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:25:05.783259 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:05.783225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"}
Jul 06 21:25:06.556460 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:06.556420    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:25:06.557761 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:06.557735    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:25:06.557876 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:06.557785    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:25:07.556834 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:07.556803    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:25:07.557248 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:07.556864    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:25:08.556820 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:08.556781    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:25:08.557038 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:08.556846    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:25:09.556049 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:09.556015    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:25:09.556226 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:09.556073    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:25:10.556258 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:10.556225    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:25:10.556671 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:10.556297    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:25:12.390119 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:12.390085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:25:12.390634 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:12.390364    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:25:12.391296 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:12.391275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:25:29.941500 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:29.941468    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653" exitCode=137
Jul 06 21:25:29.942078 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:29.941535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"}
Jul 06 21:25:29.942078 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:29.941600    2582 scope.go:117] "RemoveContainer" containerID="3de3289c8d81718a7c3800f675d069be5baae55d77890c9f90d47f2d1b96ff7e"
Jul 06 21:25:29.942207 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:29.942120    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:25:29.942394 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:25:29.942366    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:25:41.331746 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:41.331706    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:25:41.332244 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:25:41.331956    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:25:42.391109 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:42.391073    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:25:42.391607 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:42.391591    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:25:42.391808 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:25:42.391784    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:25:46.726209 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:46.726173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:25:53.335272 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:25:53.335244    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:25:53.335718 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:25:53.335452    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:26:08.331758 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:26:08.331668    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:26:08.332452 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:26:08.331940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:26:23.335994 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:26:23.335950    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:26:23.336407 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:26:23.336171    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:26:35.331177 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:26:35.331146    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:26:35.331580 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:26:35.331361    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:26:38.555055 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:26:38.555027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:26:47.331727 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:26:47.331693    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:26:47.332232 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:26:47.331859    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:26:58.331716 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:26:58.331684    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:26:58.332193 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:26:58.331873    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:27:12.331361 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:27:12.331324    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:27:12.331839 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:27:12.331534    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:27:25.331494 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:27:25.331413    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:27:25.331981 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:27:25.331605    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:27:28.190255 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:27:28.190224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:27:37.330896 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:27:37.330865    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:27:37.331345 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:27:37.331101    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:27:49.331139 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:27:49.331107    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:27:49.331630 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:27:49.331309    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:28:00.331043 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:00.331014    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:28:00.331592 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:28:00.331279    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:28:12.331460 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:12.331428    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:28:12.332561 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:12.332530    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:28:12.332626 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:12.332598    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:28:12.332683 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:12.332633    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:28:12.915851 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:12.915807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"}
Jul 06 21:28:13.556299 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:13.556255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:28:13.557475 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:13.557449    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:28:13.557612 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:13.557510    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:28:14.556244 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:14.556206    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:28:14.556421 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:14.556273    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:28:15.556117 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:15.556080    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:28:15.556322 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:15.556164    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:28:16.556183 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:16.556142    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:28:16.556560 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:16.556220    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:28:17.556213 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:17.556175    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:28:17.556623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:17.556242    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:28:18.556391 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:18.556353    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:28:18.556782 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:18.556425    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:28:20.430225 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:20.430191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:28:20.430675 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:20.430502    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:28:20.431510 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:20.431491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:28:38.077474 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:38.077437    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d" exitCode=137
Jul 06 21:28:38.077927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:38.077513    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"}
Jul 06 21:28:38.077927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:38.077588    2582 scope.go:117] "RemoveContainer" containerID="02d355766d21fe0ce3a71c4fde6976a82c60c2193fdcdd553c12ed9472591653"
Jul 06 21:28:38.078047 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:38.077951    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:28:38.078201 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:28:38.078181    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:28:50.331539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:50.331504    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:28:50.331957 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:28:50.331697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:28:50.430755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:50.430717    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:28:51.156576 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:51.156548    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:28:51.156757 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:28:51.156734    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:28:52.926453 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:28:52.926416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:29:03.334609 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:29:03.334532    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:29:03.335003 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:29:03.334773    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:29:04.697944 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:29:04.697908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:29:14.331559 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:29:14.331525    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:29:14.332030 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:29:14.331774    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:29:28.330733 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:29:28.330700    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:29:28.331157 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:29:28.330918    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:29:43.335408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:29:43.335379    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:29:43.335795 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:29:43.335523    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:29:57.331277 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:29:57.331245    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:29:57.331741 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:29:57.331438    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:30:03.566656 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:30:03.566628    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:30:03.581759 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:30:03.581725    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:30:12.331224 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:30:12.331191    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:30:12.331625 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:30:12.331382    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:30:24.330941 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:30:24.330911    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:30:24.331394 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:30:24.331149    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:30:38.331563 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:30:38.331529    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:30:38.332082 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:30:38.331706    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:30:52.331072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:30:52.331041    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:30:52.331460 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:30:52.331244    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:31:06.331208 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:31:06.331177    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:31:06.331622 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:31:06.331348    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:31:21.330800 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:31:21.330764    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:31:21.331322 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:31:21.330954    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:31:36.331482 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:31:36.331452    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:31:36.331950 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:31:36.331649    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:31:48.331661 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:31:48.331628    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:31:48.332162 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:31:48.331853    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:32:02.331013 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:32:02.330901    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:32:02.331508 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:32:02.331115    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:32:17.331427 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:32:17.331388    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:32:17.331901 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:32:17.331653    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:32:22.752177 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:32:22.752140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:32:28.331589 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:32:28.331557    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:32:28.332000 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:32:28.331739    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:32:42.331719 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:32:42.331686    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:32:42.332174 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:32:42.331886    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:32:55.331713 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:32:55.331677    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:32:55.332329 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:32:55.331893    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:33:07.331423 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:07.331379    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:33:07.332338 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:33:07.331615    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:33:21.331789 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:21.331754    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:33:21.332439 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:33:21.331958    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:33:32.331113 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:32.331074    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:33:32.333636 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:33:32.331259    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:33:44.331752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:44.331721    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:33:44.332752 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:44.332720    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:33:44.332862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:44.332803    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:33:44.332862 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:44.332853    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:33:44.867848 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:44.867815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"}
Jul 06 21:33:45.556042 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:45.556007    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:33:45.557264 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:45.557237    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:33:45.557404 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:45.557294    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:33:46.556231 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:46.556195    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:33:46.556610 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:46.556251    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:33:47.556751 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:47.556718    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:33:47.557164 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:47.556771    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:33:48.556553 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:48.556516    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:33:48.556737 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:48.556576    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:33:49.556480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:49.556446    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:33:49.556941 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:49.556566    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:33:50.431067 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:50.431031    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:33:50.560116 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:50.560083    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:33:50.905302 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:33:50.905223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:34:47.230207 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:34:47.230157    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"]
Jul 06 21:34:47.259878 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:34:47.259813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:34:49.097286 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:34:49.097251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:35:02.259365 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:02.259308    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main" containerID="cri-o://1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9" gracePeriod=45
Jul 06 21:35:02.931557 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:02.931529    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:35:03.105755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tls-certs\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.105755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-home\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.106058 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105763    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kserve-provision-location\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.106058 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105783    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tmp-dir\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.106058 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105810    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-dshm\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.106058 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-model-cache\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.106058 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8dsv\" (UniqueName: \"kubernetes.io/projected/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kube-api-access-m8dsv\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.106058 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.105941    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5af5fefd-2f75-44ea-9a08-4424e3c06b67-cabundle-cert\") pod \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\" (UID: \"5af5fefd-2f75-44ea-9a08-4424e3c06b67\") "
Jul 06 21:35:03.106385 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.106148    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-model-cache" (OuterVolumeSpecName: "model-cache") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:03.106385 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.106212    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-home" (OuterVolumeSpecName: "home") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:03.106385 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.106381    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.106544 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.106399    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.106544 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.106427    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af5fefd-2f75-44ea-9a08-4424e3c06b67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:35:03.108101 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.108063    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:35:03.108194 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.108159    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kube-api-access-m8dsv" (OuterVolumeSpecName: "kube-api-access-m8dsv") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "kube-api-access-m8dsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:35:03.108261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.108199    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-dshm" (OuterVolumeSpecName: "dshm") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:03.124129 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.124100    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:03.163186 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.163155    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5af5fefd-2f75-44ea-9a08-4424e3c06b67" (UID: "5af5fefd-2f75-44ea-9a08-4424e3c06b67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:35:03.207760 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.207735    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dsv\" (UniqueName: \"kubernetes.io/projected/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kube-api-access-m8dsv\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.207760 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.207759    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5af5fefd-2f75-44ea-9a08-4424e3c06b67-cabundle-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.207760 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.207768    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.207986 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.207775    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.207986 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.207784    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.207986 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.207792    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af5fefd-2f75-44ea-9a08-4424e3c06b67-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:35:03.359783 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.359711    2582 generic.go:356] "Generic (PLEG): container finished" podID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerID="1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9" exitCode=0
Jul 06 21:35:03.359783 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.359763    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"
Jul 06 21:35:03.360206 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.359790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" event={"ID":"5af5fefd-2f75-44ea-9a08-4424e3c06b67","Type":"ContainerDied","Data":"1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9"}
Jul 06 21:35:03.360206 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.359842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp" event={"ID":"5af5fefd-2f75-44ea-9a08-4424e3c06b67","Type":"ContainerDied","Data":"c71ae939a1e3d06d77a45080e7ac92453de5a90bb1d2bf72420878db637f1389"}
Jul 06 21:35:03.360206 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.359864    2582 scope.go:117] "RemoveContainer" containerID="1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9"
Jul 06 21:35:03.368993 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.368973    2582 scope.go:117] "RemoveContainer" containerID="75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9"
Jul 06 21:35:03.377446 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.377429    2582 scope.go:117] "RemoveContainer" containerID="1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9"
Jul 06 21:35:03.377714 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:35:03.377692    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9\": container with ID starting with 1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9 not found: ID does not exist" containerID="1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9"
Jul 06 21:35:03.377766 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.377722    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9"} err="failed to get container status \"1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9\": rpc error: code = NotFound desc = could not find container \"1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9\": container with ID starting with 1884d44a6f607c0fb0df800412ef46e976b3db732b74f2924207c690240f33b9 not found: ID does not exist"
Jul 06 21:35:03.377766 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.377745    2582 scope.go:117] "RemoveContainer" containerID="75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9"
Jul 06 21:35:03.377991 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:35:03.377954    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9\": container with ID starting with 75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9 not found: ID does not exist" containerID="75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9"
Jul 06 21:35:03.378094 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.378003    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9"} err="failed to get container status \"75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9\": rpc error: code = NotFound desc = could not find container \"75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9\": container with ID starting with 75e941d96ece48f7ce0aaa283f8db06ee2afeea951d64bc5259462edd75dc9b9 not found: ID does not exist"
Jul 06 21:35:03.381997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.381956    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"]
Jul 06 21:35:03.387248 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.387227    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp"]
Jul 06 21:35:03.616582 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.616504    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:35:03.626481 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:03.626452    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:35:05.337654 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:05.337622    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" path="/var/lib/kubelet/pods/5af5fefd-2f75-44ea-9a08-4424e3c06b67/volumes"
Jul 06 21:35:17.161005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.160889    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"]
Jul 06 21:35:17.161567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.161456    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main"
Jul 06 21:35:17.161567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.161473    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main"
Jul 06 21:35:17.161567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.161501    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="storage-initializer"
Jul 06 21:35:17.161567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.161507    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="storage-initializer"
Jul 06 21:35:17.161781 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.161581    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af5fefd-2f75-44ea-9a08-4424e3c06b67" containerName="main"
Jul 06 21:35:17.167234 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.167213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.168072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.168052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:35:17.170093 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.170074    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 06 21:35:17.170182 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.170074    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:35:17.183813 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.183790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"]
Jul 06 21:35:17.237901 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.237863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.238160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.237918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-home\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.238160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.237946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.238160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.238030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-dshm\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.238160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.238052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dws\" (UniqueName: \"kubernetes.io/projected/d4948f86-f0d1-4911-908f-7cb88db94468-kube-api-access-n5dws\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.238160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.238133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-model-cache\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.238160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.238151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4948f86-f0d1-4911-908f-7cb88db94468-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.238384 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.238186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4948f86-f0d1-4911-908f-7cb88db94468-tls-certs\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.339622 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-home\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.339622 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.339899 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-dshm\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.339899 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dws\" (UniqueName: \"kubernetes.io/projected/d4948f86-f0d1-4911-908f-7cb88db94468-kube-api-access-n5dws\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.339899 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-model-cache\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.339899 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4948f86-f0d1-4911-908f-7cb88db94468-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.339899 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4948f86-f0d1-4911-908f-7cb88db94468-tls-certs\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.340257 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.339998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.340257 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.340086    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.340257 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.340116    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-home\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.340412 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.340364    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-model-cache\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.340478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.340457    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.340544 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.340531    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4948f86-f0d1-4911-908f-7cb88db94468-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.342072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.342049    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-dshm\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.342552 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.342533    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4948f86-f0d1-4911-908f-7cb88db94468-tls-certs\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.348201 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.348179    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dws\" (UniqueName: \"kubernetes.io/projected/d4948f86-f0d1-4911-908f-7cb88db94468-kube-api-access-n5dws\") pod \"lora-multiple-adapters-test-kserve-77485bff95-2tx5t\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.478444 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.478401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:17.653474 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.653442    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"]
Jul 06 21:35:17.653816 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:35:17.653792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4948f86_f0d1_4911_908f_7cb88db94468.slice/crio-299d3e66b7e1cca6fc48a1997ee24266ef60d8c156d947ca88279100e74760b7 WatchSource:0}: Error finding container 299d3e66b7e1cca6fc48a1997ee24266ef60d8c156d947ca88279100e74760b7: Status 404 returned error can't find the container with id 299d3e66b7e1cca6fc48a1997ee24266ef60d8c156d947ca88279100e74760b7
Jul 06 21:35:17.658486 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:17.658463    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:35:18.457077 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:18.457038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" event={"ID":"d4948f86-f0d1-4911-908f-7cb88db94468","Type":"ContainerStarted","Data":"fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31"}
Jul 06 21:35:18.457499 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:18.457112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" event={"ID":"d4948f86-f0d1-4911-908f-7cb88db94468","Type":"ContainerStarted","Data":"299d3e66b7e1cca6fc48a1997ee24266ef60d8c156d947ca88279100e74760b7"}
Jul 06 21:35:23.487554 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:23.487517    2582 generic.go:356] "Generic (PLEG): container finished" podID="d4948f86-f0d1-4911-908f-7cb88db94468" containerID="fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31" exitCode=0
Jul 06 21:35:23.487945 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:23.487590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" event={"ID":"d4948f86-f0d1-4911-908f-7cb88db94468","Type":"ContainerDied","Data":"fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31"}
Jul 06 21:35:24.497387 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:24.497335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" event={"ID":"d4948f86-f0d1-4911-908f-7cb88db94468","Type":"ContainerStarted","Data":"d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402"}
Jul 06 21:35:24.525976 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:24.525912    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podStartSLOduration=2.484348133 podStartE2EDuration="7.52589374s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.041545607s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:35:24.522589707 +0000 UTC m=+3621.856930321" watchObservedRunningTime="2026-07-06 21:35:24.52589374 +0000 UTC m=+3621.860234336"
Jul 06 21:35:25.479046 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:25.478994    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:27.479174 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:27.479131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:35:27.481409 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:27.481373    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:35:37.479494 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:37.479453    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:35:42.651512 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:42.651480    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f" exitCode=137
Jul 06 21:35:42.652006 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:42.651549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"}
Jul 06 21:35:42.652006 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:42.651623    2582 scope.go:117] "RemoveContainer" containerID="8dfa53b74a50decf7cc46fae4be6f9be9e21e5bf0364a75eb86ec3a720aff49d"
Jul 06 21:35:42.652200 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:42.652171    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:35:42.652488 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:35:42.652459    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:35:47.479364 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:47.479313    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:35:50.904319 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:50.904288    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:35:50.904837 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:50.904820    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:35:50.905075 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:35:50.905055    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:35:57.479255 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:35:57.479201    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:36:02.894002 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:02.893938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:36:03.335080 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:03.335053    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:36:03.335586 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:36:03.335553    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:36:07.479910 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:07.479855    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:36:17.331078 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:17.331038    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:36:17.331549 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:36:17.331250    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:36:17.479749 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:17.479708    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:36:27.479526 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:27.479414    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:36:32.331445 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:32.331402    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:36:32.331938 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:36:32.331595    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:36:37.479763 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:37.479699    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:36:43.335861 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:43.335835    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:36:43.336278 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:36:43.336047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:36:47.490019 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:47.489985    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:36:48.083587 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:48.083552    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:36:57.331223 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:36:57.331193    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:36:57.331636 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:36:57.331433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:37:11.331020 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:37:11.330989    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:37:11.331498 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:37:11.331191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:37:22.330955 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:37:22.330922    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:37:22.331402 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:37:22.331137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:37:37.331259 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:37:37.331217    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:37:37.331870 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:37:37.331491    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:37:49.948917 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:37:49.948885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:37:52.331327 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:37:52.331296    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:37:52.331738 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:37:52.331485    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:38:07.330810 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:38:07.330737    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:38:07.331232 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:38:07.330982    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:38:20.331476 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:38:20.331437    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:38:20.332107 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:38:20.331653    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:38:32.330922 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:38:32.330891    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:38:32.331524 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:38:32.331122    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:38:46.331357 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:38:46.331324    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:38:46.331775 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:38:46.331535    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:39:01.331009 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:39:01.330953    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:39:01.331614 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:39:01.331147    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:39:12.331535 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:39:12.331506    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:39:12.331954 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:39:12.331702    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:39:24.331662 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:39:24.331631    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:39:24.332056 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:39:24.331807    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:39:36.331235 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:39:36.331194    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:39:36.331804 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:39:36.331447    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:39:51.331409 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:39:51.331377    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:39:51.331860 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:39:51.331563    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:40:03.335456 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:03.335424    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:40:03.335918 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:40:03.335652    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:40:03.655451 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:03.655372    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:40:03.665768 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:03.665743    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:40:17.331182 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:17.331145    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:40:17.331783 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:40:17.331383    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:40:29.331075 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:29.331047    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:40:29.331499 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:40:29.331246    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:40:43.335672 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:43.335635    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:40:43.336753 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:43.336720    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:40:43.336884 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:43.336790    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:40:43.336884 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:43.336822    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:40:43.479434 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:43.479397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"}
Jul 06 21:40:43.556072 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:43.556036    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:40:44.557322 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:44.557291    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:40:44.557771 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:44.557344    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:40:45.556766 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:45.556731    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:40:45.556945 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:45.556788    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:40:46.556358 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:46.556328    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:40:46.556735 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:46.556379    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:40:47.556256 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:47.556218    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:40:47.556444 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:47.556300    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:40:48.555822 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:48.555788    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:40:48.556104 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:48.555842    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:40:49.429284 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:49.429245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:40:49.556480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:49.556443    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:40:49.556691 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:49.556531    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:40:50.904875 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:50.904836    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:40:51.598133 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:51.598091    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 06 21:40:51.598346 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:51.598154    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:40:52.000057 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:52.000023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:40:52.001375 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:40:52.001357    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:41:05.074527 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:05.074496    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:05.102315 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:05.102279    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"]
Jul 06 21:41:05.106856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:05.106828    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"]
Jul 06 21:41:05.148923 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:05.148892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:09.678910 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:09.678875    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6" exitCode=137
Jul 06 21:41:09.679413 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:09.678941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"}
Jul 06 21:41:09.679413 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:09.679018    2582 scope.go:117] "RemoveContainer" containerID="4319d73a6ff8c624a273bc11e0133ec64c23905e07d9ca8b92da2c9d900ee44f"
Jul 06 21:41:09.679542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:09.679518    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:41:09.679791 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:09.679768    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:41:20.128538 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:20.128472    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main" containerID="cri-o://2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8" gracePeriod=45
Jul 06 21:41:20.130604 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:20.130572    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main" containerID="cri-o://76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f" gracePeriod=45
Jul 06 21:41:21.331703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.331672    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:41:21.332185 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:21.331928    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:41:21.342755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.342720    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="llm-d-routing-sidecar" containerID="cri-o://5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe" gracePeriod=44
Jul 06 21:41:21.464816 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.464790    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:41:21.523711 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.523685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-kserve-provision-location\") pod \"83e3fa00-b567-4615-ab78-00027c8a901f\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") "
Jul 06 21:41:21.523871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.523732    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-tmp-dir\") pod \"83e3fa00-b567-4615-ab78-00027c8a901f\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") "
Jul 06 21:41:21.523871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.523781    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-model-cache\") pod \"83e3fa00-b567-4615-ab78-00027c8a901f\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") "
Jul 06 21:41:21.523871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.523830    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-dshm\") pod \"83e3fa00-b567-4615-ab78-00027c8a901f\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") "
Jul 06 21:41:21.523871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.523864    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-home\") pod \"83e3fa00-b567-4615-ab78-00027c8a901f\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") "
Jul 06 21:41:21.524124 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.523908    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7w8k\" (UniqueName: \"kubernetes.io/projected/83e3fa00-b567-4615-ab78-00027c8a901f-kube-api-access-q7w8k\") pod \"83e3fa00-b567-4615-ab78-00027c8a901f\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") "
Jul 06 21:41:21.524124 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.523954    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e3fa00-b567-4615-ab78-00027c8a901f-tls-certs\") pod \"83e3fa00-b567-4615-ab78-00027c8a901f\" (UID: \"83e3fa00-b567-4615-ab78-00027c8a901f\") "
Jul 06 21:41:21.524124 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.524055    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-model-cache" (OuterVolumeSpecName: "model-cache") pod "83e3fa00-b567-4615-ab78-00027c8a901f" (UID: "83e3fa00-b567-4615-ab78-00027c8a901f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.524311 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.524273    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.525106 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.525061    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-home" (OuterVolumeSpecName: "home") pod "83e3fa00-b567-4615-ab78-00027c8a901f" (UID: "83e3fa00-b567-4615-ab78-00027c8a901f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.527517 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.527493    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e3fa00-b567-4615-ab78-00027c8a901f-kube-api-access-q7w8k" (OuterVolumeSpecName: "kube-api-access-q7w8k") pod "83e3fa00-b567-4615-ab78-00027c8a901f" (UID: "83e3fa00-b567-4615-ab78-00027c8a901f"). InnerVolumeSpecName "kube-api-access-q7w8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:41:21.528568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.528543    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e3fa00-b567-4615-ab78-00027c8a901f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83e3fa00-b567-4615-ab78-00027c8a901f" (UID: "83e3fa00-b567-4615-ab78-00027c8a901f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:41:21.528735 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.528709    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-dshm" (OuterVolumeSpecName: "dshm") pod "83e3fa00-b567-4615-ab78-00027c8a901f" (UID: "83e3fa00-b567-4615-ab78-00027c8a901f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.543660 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.543544    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83e3fa00-b567-4615-ab78-00027c8a901f" (UID: "83e3fa00-b567-4615-ab78-00027c8a901f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.602635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.602602    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83e3fa00-b567-4615-ab78-00027c8a901f" (UID: "83e3fa00-b567-4615-ab78-00027c8a901f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.606094 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.606071    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:41:21.624975 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.624933    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjmbf\" (UniqueName: \"kubernetes.io/projected/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kube-api-access-vjmbf\") pod \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") "
Jul 06 21:41:21.625101 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-model-cache\") pod \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") "
Jul 06 21:41:21.625101 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tmp-dir\") pod \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") "
Jul 06 21:41:21.625201 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625120    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-home\") pod \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") "
Jul 06 21:41:21.625201 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625149    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tls-certs\") pod \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") "
Jul 06 21:41:21.625201 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-dshm\") pod \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") "
Jul 06 21:41:21.625369 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625249    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kserve-provision-location\") pod \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\" (UID: \"3dcd0539-2d8d-4321-b366-da5555bf3fe1\") "
Jul 06 21:41:21.625369 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625290    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-model-cache" (OuterVolumeSpecName: "model-cache") pod "3dcd0539-2d8d-4321-b366-da5555bf3fe1" (UID: "3dcd0539-2d8d-4321-b366-da5555bf3fe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.625585 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625562    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.625678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625589    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.625678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625602    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.625678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625616    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7w8k\" (UniqueName: \"kubernetes.io/projected/83e3fa00-b567-4615-ab78-00027c8a901f-kube-api-access-q7w8k\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.625678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625629    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e3fa00-b567-4615-ab78-00027c8a901f-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.625678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625643    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.625678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625656    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e3fa00-b567-4615-ab78-00027c8a901f-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.625953 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.625794    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-home" (OuterVolumeSpecName: "home") pod "3dcd0539-2d8d-4321-b366-da5555bf3fe1" (UID: "3dcd0539-2d8d-4321-b366-da5555bf3fe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.627702 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.627673    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kube-api-access-vjmbf" (OuterVolumeSpecName: "kube-api-access-vjmbf") pod "3dcd0539-2d8d-4321-b366-da5555bf3fe1" (UID: "3dcd0539-2d8d-4321-b366-da5555bf3fe1"). InnerVolumeSpecName "kube-api-access-vjmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:41:21.627820 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.627682    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3dcd0539-2d8d-4321-b366-da5555bf3fe1" (UID: "3dcd0539-2d8d-4321-b366-da5555bf3fe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:41:21.627891 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.627829    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-dshm" (OuterVolumeSpecName: "dshm") pod "3dcd0539-2d8d-4321-b366-da5555bf3fe1" (UID: "3dcd0539-2d8d-4321-b366-da5555bf3fe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.645787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.645747    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3dcd0539-2d8d-4321-b366-da5555bf3fe1" (UID: "3dcd0539-2d8d-4321-b366-da5555bf3fe1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.647270 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.647241    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3dcd0539-2d8d-4321-b366-da5555bf3fe1" (UID: "3dcd0539-2d8d-4321-b366-da5555bf3fe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:41:21.727116 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.727080    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.727116 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.727108    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.727116 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.727116    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcd0539-2d8d-4321-b366-da5555bf3fe1-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.727116 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.727124    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.727388 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.727133    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.727388 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.727143    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmbf\" (UniqueName: \"kubernetes.io/projected/3dcd0539-2d8d-4321-b366-da5555bf3fe1-kube-api-access-vjmbf\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:41:21.757617 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.757574    2582 generic.go:356] "Generic (PLEG): container finished" podID="83e3fa00-b567-4615-ab78-00027c8a901f" containerID="2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8" exitCode=0
Jul 06 21:41:21.757617 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.757613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" event={"ID":"83e3fa00-b567-4615-ab78-00027c8a901f","Type":"ContainerDied","Data":"2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8"}
Jul 06 21:41:21.757856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.757626    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"
Jul 06 21:41:21.757856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.757670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l" event={"ID":"83e3fa00-b567-4615-ab78-00027c8a901f","Type":"ContainerDied","Data":"45bfc66c7c3682f855da3f34bddaba2fc66792712095a747483fd96d9f73e1f5"}
Jul 06 21:41:21.757856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.757696    2582 scope.go:117] "RemoveContainer" containerID="2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8"
Jul 06 21:41:21.760071 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.760035    2582 generic.go:356] "Generic (PLEG): container finished" podID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerID="76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f" exitCode=0
Jul 06 21:41:21.760071 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.760059    2582 generic.go:356] "Generic (PLEG): container finished" podID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerID="5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe" exitCode=0
Jul 06 21:41:21.760260 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.760095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerDied","Data":"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"}
Jul 06 21:41:21.760260 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.760139    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"
Jul 06 21:41:21.760260 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.760141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerDied","Data":"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"}
Jul 06 21:41:21.760260 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.760259    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l" event={"ID":"3dcd0539-2d8d-4321-b366-da5555bf3fe1","Type":"ContainerDied","Data":"7faa85665d8c7166b48f772bfe1a2a1f973f193258de9d8edc9483757eb6ebe0"}
Jul 06 21:41:21.768041 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.768025    2582 scope.go:117] "RemoveContainer" containerID="5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b"
Jul 06 21:41:21.777207 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.777191    2582 scope.go:117] "RemoveContainer" containerID="2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8"
Jul 06 21:41:21.777469 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:21.777445    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8\": container with ID starting with 2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8 not found: ID does not exist" containerID="2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8"
Jul 06 21:41:21.777534 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.777480    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8"} err="failed to get container status \"2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8\": rpc error: code = NotFound desc = could not find container \"2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8\": container with ID starting with 2e6dc12452eff5d74bf8f636ae43be2bb91b5618a8f3a70ff0198ca0eb607ac8 not found: ID does not exist"
Jul 06 21:41:21.777534 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.777503    2582 scope.go:117] "RemoveContainer" containerID="5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b"
Jul 06 21:41:21.777741 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:21.777724    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b\": container with ID starting with 5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b not found: ID does not exist" containerID="5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b"
Jul 06 21:41:21.777813 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.777749    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b"} err="failed to get container status \"5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b\": rpc error: code = NotFound desc = could not find container \"5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b\": container with ID starting with 5837d6cd4103179de8cb7706f632accb33b5c19124f713770fab1581cd79234b not found: ID does not exist"
Jul 06 21:41:21.777813 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.777770    2582 scope.go:117] "RemoveContainer" containerID="76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"
Jul 06 21:41:21.784854 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.784830    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"]
Jul 06 21:41:21.787377 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.787354    2582 scope.go:117] "RemoveContainer" containerID="a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"
Jul 06 21:41:21.788755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.788737    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l"]
Jul 06 21:41:21.797910 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.797890    2582 scope.go:117] "RemoveContainer" containerID="5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"
Jul 06 21:41:21.801678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.801655    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"]
Jul 06 21:41:21.805402 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.805382    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l"]
Jul 06 21:41:21.807563 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.807546    2582 scope.go:117] "RemoveContainer" containerID="76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"
Jul 06 21:41:21.807899 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:21.807868    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f\": container with ID starting with 76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f not found: ID does not exist" containerID="76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"
Jul 06 21:41:21.808034 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.807905    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"} err="failed to get container status \"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f\": rpc error: code = NotFound desc = could not find container \"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f\": container with ID starting with 76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f not found: ID does not exist"
Jul 06 21:41:21.808034 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.807932    2582 scope.go:117] "RemoveContainer" containerID="a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"
Jul 06 21:41:21.808311 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:21.808289    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f\": container with ID starting with a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f not found: ID does not exist" containerID="a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"
Jul 06 21:41:21.808390 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.808317    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"} err="failed to get container status \"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f\": rpc error: code = NotFound desc = could not find container \"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f\": container with ID starting with a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f not found: ID does not exist"
Jul 06 21:41:21.808390 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.808335    2582 scope.go:117] "RemoveContainer" containerID="5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"
Jul 06 21:41:21.808601 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:21.808577    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe\": container with ID starting with 5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe not found: ID does not exist" containerID="5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"
Jul 06 21:41:21.808683 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.808609    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"} err="failed to get container status \"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe\": rpc error: code = NotFound desc = could not find container \"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe\": container with ID starting with 5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe not found: ID does not exist"
Jul 06 21:41:21.808683 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.808631    2582 scope.go:117] "RemoveContainer" containerID="76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"
Jul 06 21:41:21.808904 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.808883    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f"} err="failed to get container status \"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f\": rpc error: code = NotFound desc = could not find container \"76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f\": container with ID starting with 76dc5721b5185125cc8efae267914171e1450c81ba246736033614c5a051318f not found: ID does not exist"
Jul 06 21:41:21.808904 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.808903    2582 scope.go:117] "RemoveContainer" containerID="a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"
Jul 06 21:41:21.809166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.809145    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f"} err="failed to get container status \"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f\": rpc error: code = NotFound desc = could not find container \"a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f\": container with ID starting with a3f21edef8cc708457b74bd5ec65a2725229a58f81d679c43631b044b3aaa79f not found: ID does not exist"
Jul 06 21:41:21.809166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.809164    2582 scope.go:117] "RemoveContainer" containerID="5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"
Jul 06 21:41:21.809406 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:21.809374    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe"} err="failed to get container status \"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe\": rpc error: code = NotFound desc = could not find container \"5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe\": container with ID starting with 5eb58610b112482d948768f69dfcb7b0f38247e14c0a69f34c66bbd22f584abe not found: ID does not exist"
Jul 06 21:41:22.000763 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:22.000713    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:41:22.001256 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:22.001235    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:41:22.001446 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:22.001423    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:41:23.338220 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:23.338186    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" path="/var/lib/kubelet/pods/3dcd0539-2d8d-4321-b366-da5555bf3fe1/volumes"
Jul 06 21:41:23.338759 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:23.338683    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" path="/var/lib/kubelet/pods/83e3fa00-b567-4615-ab78-00027c8a901f/volumes"
Jul 06 21:41:25.925136 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:25.925106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:26.026089 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:26.026049    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:29.597930 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:29.597894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:35.331324 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:35.331295    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:41:35.331826 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:35.331499    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.711182    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"]
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712172    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="llm-d-routing-sidecar"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712195    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="llm-d-routing-sidecar"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712232    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="storage-initializer"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712241    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="storage-initializer"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712266    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712285    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712305    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712313    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712351    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="storage-initializer"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712360    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="storage-initializer"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712566    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="main"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712589    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e3fa00-b567-4615-ab78-00027c8a901f" containerName="main"
Jul 06 21:41:44.713985 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.712609    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dcd0539-2d8d-4321-b366-da5555bf3fe1" containerName="llm-d-routing-sidecar"
Jul 06 21:41:44.722427 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.722396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.727429 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.727392    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-qwgp5\""
Jul 06 21:41:44.727586 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.727433    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 06 21:41:44.735408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.735383    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"]
Jul 06 21:41:44.739254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.739233    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.759934 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.759901    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"]
Jul 06 21:41:44.765843 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.765815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"]
Jul 06 21:41:44.769046 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.769023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:44.846771 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.846733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-model-cache\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.847015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.846781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.847015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.846803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tmp-dir\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.847015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.846899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9t5\" (UniqueName: \"kubernetes.io/projected/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kube-api-access-jz9t5\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.847015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.846946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.847015 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-home\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-home\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tls-certs\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbwj\" (UniqueName: \"kubernetes.io/projected/97118a8a-4346-4bd7-bac3-d3335301a659-kube-api-access-5cbwj\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97118a8a-4346-4bd7-bac3-d3335301a659-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-dshm\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.847280 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.847273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.948419 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-dshm\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.948419 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-model-cache\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tmp-dir\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9t5\" (UniqueName: \"kubernetes.io/projected/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kube-api-access-jz9t5\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-home\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-home\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.948703 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tls-certs\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.949171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948840    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948859    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.948920    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-home\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbwj\" (UniqueName: \"kubernetes.io/projected/97118a8a-4346-4bd7-bac3-d3335301a659-kube-api-access-5cbwj\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97118a8a-4346-4bd7-bac3-d3335301a659-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949530 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949431    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.949684 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949662    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-model-cache\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.949817 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949790    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-home\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.949882 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.949856    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tmp-dir\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.950225 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.950198    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.951364 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.951340    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tls-certs\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.951572 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.951541    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.951732 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.951714    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-dshm\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:44.951958 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.951937    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97118a8a-4346-4bd7-bac3-d3335301a659-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.956740 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.956721    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbwj\" (UniqueName: \"kubernetes.io/projected/97118a8a-4346-4bd7-bac3-d3335301a659-kube-api-access-5cbwj\") pod \"router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:44.957034 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:44.957014    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9t5\" (UniqueName: \"kubernetes.io/projected/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kube-api-access-jz9t5\") pod \"router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:45.035931 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.035832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:45.050213 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.050175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:45.196053 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.196022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"]
Jul 06 21:41:45.219261 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:41:45.219206    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97118a8a_4346_4bd7_bac3_d3335301a659.slice/crio-e259eca8dd145e67278f3661adab7d0111a6f657a37b6784b536940c41f6eed5 WatchSource:0}: Error finding container e259eca8dd145e67278f3661adab7d0111a6f657a37b6784b536940c41f6eed5: Status 404 returned error can't find the container with id e259eca8dd145e67278f3661adab7d0111a6f657a37b6784b536940c41f6eed5
Jul 06 21:41:45.219930 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.219901    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"]
Jul 06 21:41:45.224410 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.224394    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:41:45.360211 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.360092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:45.459898 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.459863    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:45.906681 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.906643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerStarted","Data":"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"}
Jul 06 21:41:45.907168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.906713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerStarted","Data":"3cf1e80cbb3104b5c081cd93a79ed8864fc895c4fb3eaad6067bb20193845e82"}
Jul 06 21:41:45.907168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.906787    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:45.908301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.908273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" event={"ID":"97118a8a-4346-4bd7-bac3-d3335301a659","Type":"ContainerStarted","Data":"f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376"}
Jul 06 21:41:45.908417 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:45.908331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" event={"ID":"97118a8a-4346-4bd7-bac3-d3335301a659","Type":"ContainerStarted","Data":"e259eca8dd145e67278f3661adab7d0111a6f657a37b6784b536940c41f6eed5"}
Jul 06 21:41:46.918110 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:46.918072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerStarted","Data":"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"}
Jul 06 21:41:48.331775 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:48.331736    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:41:48.332309 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:41:48.332051    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:41:49.947814 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:49.947778    2582 generic.go:356] "Generic (PLEG): container finished" podID="97118a8a-4346-4bd7-bac3-d3335301a659" containerID="f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376" exitCode=0
Jul 06 21:41:49.948246 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:49.947842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" event={"ID":"97118a8a-4346-4bd7-bac3-d3335301a659","Type":"ContainerDied","Data":"f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376"}
Jul 06 21:41:50.957028 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:50.956986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" event={"ID":"97118a8a-4346-4bd7-bac3-d3335301a659","Type":"ContainerStarted","Data":"7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34"}
Jul 06 21:41:50.989309 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:50.989253    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podStartSLOduration=2.595114767 podStartE2EDuration="6.989234879s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.394120112s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:41:50.988059857 +0000 UTC m=+4008.322400453" watchObservedRunningTime="2026-07-06 21:41:50.989234879 +0000 UTC m=+4008.323575474"
Jul 06 21:41:51.964438 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:51.964403    2582 generic.go:356] "Generic (PLEG): container finished" podID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerID="ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4" exitCode=0
Jul 06 21:41:51.964837 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:51.964458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerDied","Data":"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"}
Jul 06 21:41:52.983438 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:52.983401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerStarted","Data":"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"}
Jul 06 21:41:52.985292 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:52.985264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:41:53.015763 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:53.015700    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podStartSLOduration=3.341369771 podStartE2EDuration="9.015677294s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.674307523s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:41:53.011010574 +0000 UTC m=+4010.345351194" watchObservedRunningTime="2026-07-06 21:41:53.015677294 +0000 UTC m=+4010.350017892"
Jul 06 21:41:55.037115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:55.037084    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:55.037694 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:55.037258    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:41:55.039005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:55.038942    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:41:55.050322 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:55.050292    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:55.050482 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:55.050350    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:41:55.051580 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:55.051549    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:41:55.051705 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:41:55.051637    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:42:03.336958 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:03.336892    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:42:03.337406 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:42:03.337194    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:42:05.036753 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:05.036713    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:42:05.051103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:05.051050    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:42:15.036807 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:15.036763    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:42:15.050887 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:15.050834    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:42:16.331244 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:16.331211    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:42:16.331683 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:42:16.331415    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:42:25.036831 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:25.036725    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:42:25.051223 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:25.051182    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:42:29.331213 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:29.331176    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:42:29.331756 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:42:29.331419    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:42:35.036935 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:35.036882    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:42:35.051427 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:35.051387    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:42:41.331094 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:41.331061    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:42:41.331481 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:42:41.331320    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:42:45.036972 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:45.036906    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:42:45.050814 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:45.050766    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:42:52.330869 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:52.330831    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:42:52.331286 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:42:52.331118    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:42:55.037188 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:55.037142    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:42:55.051183 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:42:55.051141    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:43:04.330833 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:04.330791    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:43:04.331405 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:43:04.331014    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:43:05.036636 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:05.036589    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:43:05.051273 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:05.051229    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:43:15.036425 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:15.036375    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:43:15.050827 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:15.050794    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:43:19.337603 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:19.337565    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:43:19.338135 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:43:19.337752    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:43:25.036490 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:25.036443    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:43:25.051146 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:25.051095    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:43:28.538617 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:28.538585    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:43:34.331465 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:34.331421    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:43:34.333226 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:43:34.331618    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:43:35.036254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:35.036201    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:43:35.050555 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:35.050507    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:43:45.037108 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:45.037063    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:43:45.051559 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:45.051518    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:43:45.331168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:45.331085    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:43:45.331313 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:43:45.331287    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:43:55.036496 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:55.036387    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:43:55.050530 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:55.050485    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:43:58.330873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:43:58.330840    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:43:58.331277 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:43:58.331058    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:44:05.036545 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:05.036491    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:44:05.050914 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:05.050867    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:44:13.338203 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:13.338159    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:44:13.338780 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:44:13.338340    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:44:15.036702 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:15.036661    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:44:15.050573 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:15.050528    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:44:25.037187 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:25.037132    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Jul 06 21:44:25.051275 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:25.051221    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 06 21:44:25.332357 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:25.332255    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:44:25.332548 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:44:25.332506    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:44:35.047030 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:35.046996    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:44:35.059350 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:35.059326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:44:35.061435 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:35.061412    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:44:35.068890 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:35.068865    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:44:37.331115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:37.331082    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:44:37.331734 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:44:37.331281    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:44:41.280767 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:41.280723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:44:41.343208 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:41.343170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:44:44.606470 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:44.606434    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"]
Jul 06 21:44:44.612553 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:44.612527    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"]
Jul 06 21:44:44.616949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:44.616918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:44:44.678774 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:44.678742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:44:48.332545 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:48.332509    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:44:48.333070 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:44:48.332922    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:44:59.633838 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:59.633763    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main" containerID="cri-o://18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3" gracePeriod=45
Jul 06 21:44:59.642653 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:44:59.642612    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main" containerID="cri-o://7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34" gracePeriod=45
Jul 06 21:45:00.331421 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.331387    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:45:00.331667 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:00.331643    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:45:00.459064 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.459018    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="llm-d-routing-sidecar" containerID="cri-o://1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10" gracePeriod=45
Jul 06 21:45:00.615997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.615958    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:45:00.661446 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97118a8a-4346-4bd7-bac3-d3335301a659-tls-certs\") pod \"97118a8a-4346-4bd7-bac3-d3335301a659\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") "
Jul 06 21:45:00.661856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661451    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-tmp-dir\") pod \"97118a8a-4346-4bd7-bac3-d3335301a659\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") "
Jul 06 21:45:00.661856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661503    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-model-cache\") pod \"97118a8a-4346-4bd7-bac3-d3335301a659\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") "
Jul 06 21:45:00.661856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661527    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-home\") pod \"97118a8a-4346-4bd7-bac3-d3335301a659\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") "
Jul 06 21:45:00.661856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661567    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-dshm\") pod \"97118a8a-4346-4bd7-bac3-d3335301a659\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") "
Jul 06 21:45:00.661856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661620    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cbwj\" (UniqueName: \"kubernetes.io/projected/97118a8a-4346-4bd7-bac3-d3335301a659-kube-api-access-5cbwj\") pod \"97118a8a-4346-4bd7-bac3-d3335301a659\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") "
Jul 06 21:45:00.661856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-kserve-provision-location\") pod \"97118a8a-4346-4bd7-bac3-d3335301a659\" (UID: \"97118a8a-4346-4bd7-bac3-d3335301a659\") "
Jul 06 21:45:00.661856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.661800    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-model-cache" (OuterVolumeSpecName: "model-cache") pod "97118a8a-4346-4bd7-bac3-d3335301a659" (UID: "97118a8a-4346-4bd7-bac3-d3335301a659"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.662237 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.662090    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.662237 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.662221    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-home" (OuterVolumeSpecName: "home") pod "97118a8a-4346-4bd7-bac3-d3335301a659" (UID: "97118a8a-4346-4bd7-bac3-d3335301a659"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.663938 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.663900    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97118a8a-4346-4bd7-bac3-d3335301a659-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97118a8a-4346-4bd7-bac3-d3335301a659" (UID: "97118a8a-4346-4bd7-bac3-d3335301a659"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:45:00.664138 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.664117    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97118a8a-4346-4bd7-bac3-d3335301a659-kube-api-access-5cbwj" (OuterVolumeSpecName: "kube-api-access-5cbwj") pod "97118a8a-4346-4bd7-bac3-d3335301a659" (UID: "97118a8a-4346-4bd7-bac3-d3335301a659"). InnerVolumeSpecName "kube-api-access-5cbwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:45:00.664354 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.664331    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-dshm" (OuterVolumeSpecName: "dshm") pod "97118a8a-4346-4bd7-bac3-d3335301a659" (UID: "97118a8a-4346-4bd7-bac3-d3335301a659"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.682030 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.681990    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97118a8a-4346-4bd7-bac3-d3335301a659" (UID: "97118a8a-4346-4bd7-bac3-d3335301a659"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.724322 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.724298    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:45:00.739055 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.739030    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97118a8a-4346-4bd7-bac3-d3335301a659" (UID: "97118a8a-4346-4bd7-bac3-d3335301a659"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.763031 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763004    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tmp-dir\") pod \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") "
Jul 06 21:45:00.763166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763039    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-home\") pod \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") "
Jul 06 21:45:00.763166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tls-certs\") pod \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") "
Jul 06 21:45:00.763166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763079    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-dshm\") pod \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") "
Jul 06 21:45:00.763166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763158    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-model-cache\") pod \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") "
Jul 06 21:45:00.763389 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763242    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kserve-provision-location\") pod \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") "
Jul 06 21:45:00.763389 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz9t5\" (UniqueName: \"kubernetes.io/projected/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kube-api-access-jz9t5\") pod \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\" (UID: \"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64\") "
Jul 06 21:45:00.763870 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763581    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.763870 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763602    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbwj\" (UniqueName: \"kubernetes.io/projected/97118a8a-4346-4bd7-bac3-d3335301a659-kube-api-access-5cbwj\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.763870 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763617    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.763870 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763631    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97118a8a-4346-4bd7-bac3-d3335301a659-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.763870 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763644    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.763870 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763656    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97118a8a-4346-4bd7-bac3-d3335301a659-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.763870 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.763756    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-home" (OuterVolumeSpecName: "home") pod "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" (UID: "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.764331 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.764206    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" (UID: "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.765443 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.765411    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-dshm" (OuterVolumeSpecName: "dshm") pod "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" (UID: "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.765552 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.765528    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kube-api-access-jz9t5" (OuterVolumeSpecName: "kube-api-access-jz9t5") pod "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" (UID: "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64"). InnerVolumeSpecName "kube-api-access-jz9t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:45:00.765655 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.765636    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" (UID: "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:45:00.785406 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.785373    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" (UID: "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.828846 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.828797    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" (UID: "7fe05fbd-2e12-44fe-9d6a-cafee21ecb64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:45:00.864480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.864386    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz9t5\" (UniqueName: \"kubernetes.io/projected/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kube-api-access-jz9t5\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.864480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.864414    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.864480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.864424    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.864480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.864432    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.864480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.864440    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.864480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.864447    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:00.864480 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:00.864460    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:45:01.208216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.208180    2582 generic.go:356] "Generic (PLEG): container finished" podID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerID="18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3" exitCode=0
Jul 06 21:45:01.208216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.208209    2582 generic.go:356] "Generic (PLEG): container finished" podID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerID="1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10" exitCode=0
Jul 06 21:45:01.208484 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.208254    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"
Jul 06 21:45:01.208484 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.208264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerDied","Data":"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"}
Jul 06 21:45:01.208484 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.208330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerDied","Data":"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"}
Jul 06 21:45:01.208484 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.208359    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47" event={"ID":"7fe05fbd-2e12-44fe-9d6a-cafee21ecb64","Type":"ContainerDied","Data":"3cf1e80cbb3104b5c081cd93a79ed8864fc895c4fb3eaad6067bb20193845e82"}
Jul 06 21:45:01.208484 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.208390    2582 scope.go:117] "RemoveContainer" containerID="18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"
Jul 06 21:45:01.210389 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.210360    2582 generic.go:356] "Generic (PLEG): container finished" podID="97118a8a-4346-4bd7-bac3-d3335301a659" containerID="7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34" exitCode=0
Jul 06 21:45:01.210517 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.210424    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"
Jul 06 21:45:01.210517 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.210442    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" event={"ID":"97118a8a-4346-4bd7-bac3-d3335301a659","Type":"ContainerDied","Data":"7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34"}
Jul 06 21:45:01.210517 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.210497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5" event={"ID":"97118a8a-4346-4bd7-bac3-d3335301a659","Type":"ContainerDied","Data":"e259eca8dd145e67278f3661adab7d0111a6f657a37b6784b536940c41f6eed5"}
Jul 06 21:45:01.227567 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.227378    2582 scope.go:117] "RemoveContainer" containerID="ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"
Jul 06 21:45:01.238159 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.238140    2582 scope.go:117] "RemoveContainer" containerID="1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"
Jul 06 21:45:01.244183 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.244157    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"]
Jul 06 21:45:01.247808 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.247776    2582 scope.go:117] "RemoveContainer" containerID="18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"
Jul 06 21:45:01.248099 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:01.248077    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3\": container with ID starting with 18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3 not found: ID does not exist" containerID="18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"
Jul 06 21:45:01.248198 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248108    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"} err="failed to get container status \"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3\": rpc error: code = NotFound desc = could not find container \"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3\": container with ID starting with 18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3 not found: ID does not exist"
Jul 06 21:45:01.248198 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248130    2582 scope.go:117] "RemoveContainer" containerID="ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"
Jul 06 21:45:01.248419 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:01.248397    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4\": container with ID starting with ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4 not found: ID does not exist" containerID="ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"
Jul 06 21:45:01.248496 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248425    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"} err="failed to get container status \"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4\": rpc error: code = NotFound desc = could not find container \"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4\": container with ID starting with ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4 not found: ID does not exist"
Jul 06 21:45:01.248496 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248442    2582 scope.go:117] "RemoveContainer" containerID="1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"
Jul 06 21:45:01.248706 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:01.248688    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10\": container with ID starting with 1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10 not found: ID does not exist" containerID="1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"
Jul 06 21:45:01.248772 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248708    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"} err="failed to get container status \"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10\": rpc error: code = NotFound desc = could not find container \"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10\": container with ID starting with 1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10 not found: ID does not exist"
Jul 06 21:45:01.248772 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248721    2582 scope.go:117] "RemoveContainer" containerID="18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"
Jul 06 21:45:01.248987 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248943    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3"} err="failed to get container status \"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3\": rpc error: code = NotFound desc = could not find container \"18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3\": container with ID starting with 18783b2205fdc364ef461c621682c4c17796efa3cddc02f87ad80bd0a2cd05b3 not found: ID does not exist"
Jul 06 21:45:01.249065 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.248987    2582 scope.go:117] "RemoveContainer" containerID="ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"
Jul 06 21:45:01.249224 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.249206    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4"} err="failed to get container status \"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4\": rpc error: code = NotFound desc = could not find container \"ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4\": container with ID starting with ce0a786c0ec07034029a55239d022311eb3156d4568f0c4f357879d5c875ddd4 not found: ID does not exist"
Jul 06 21:45:01.249286 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.249225    2582 scope.go:117] "RemoveContainer" containerID="1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"
Jul 06 21:45:01.249424 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.249407    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10"} err="failed to get container status \"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10\": rpc error: code = NotFound desc = could not find container \"1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10\": container with ID starting with 1ef21f7fba2a135dd650825f8a50b925928db310e825fd21d89ec6e8c9498f10 not found: ID does not exist"
Jul 06 21:45:01.249424 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.249423    2582 scope.go:117] "RemoveContainer" containerID="7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34"
Jul 06 21:45:01.249640 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.249620    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47"]
Jul 06 21:45:01.258772 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.258752    2582 scope.go:117] "RemoveContainer" containerID="f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376"
Jul 06 21:45:01.261304 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.261285    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"]
Jul 06 21:45:01.266096 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.266072    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5"]
Jul 06 21:45:01.268291 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.268274    2582 scope.go:117] "RemoveContainer" containerID="7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34"
Jul 06 21:45:01.268517 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:01.268497    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34\": container with ID starting with 7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34 not found: ID does not exist" containerID="7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34"
Jul 06 21:45:01.268564 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.268525    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34"} err="failed to get container status \"7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34\": rpc error: code = NotFound desc = could not find container \"7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34\": container with ID starting with 7fb94bc5bd87f75edf64ad2889ba3d1d6e2cad6228b4f4f8ef9c2300799e8d34 not found: ID does not exist"
Jul 06 21:45:01.268564 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.268544    2582 scope.go:117] "RemoveContainer" containerID="f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376"
Jul 06 21:45:01.268822 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:01.268803    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376\": container with ID starting with f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376 not found: ID does not exist" containerID="f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376"
Jul 06 21:45:01.268878 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.268827    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376"} err="failed to get container status \"f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376\": rpc error: code = NotFound desc = could not find container \"f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376\": container with ID starting with f7b5a696495c0dab64230d5d33e770f700758d12fed3106c417654104fbcd376 not found: ID does not exist"
Jul 06 21:45:01.337926 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.337892    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" path="/var/lib/kubelet/pods/7fe05fbd-2e12-44fe-9d6a-cafee21ecb64/volumes"
Jul 06 21:45:01.338478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:01.338462    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" path="/var/lib/kubelet/pods/97118a8a-4346-4bd7-bac3-d3335301a659/volumes"
Jul 06 21:45:03.691785 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:03.691756    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:45:03.704891 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:03.704867    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:45:11.329078 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:11.329045    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:45:14.264907 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.264874    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"]
Jul 06 21:45:14.265309 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265292    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main"
Jul 06 21:45:14.265309 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265305    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265314    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265320    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265338    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="storage-initializer"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265344    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="storage-initializer"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265352    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="llm-d-routing-sidecar"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265357    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="llm-d-routing-sidecar"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265364    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="storage-initializer"
Jul 06 21:45:14.265381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265369    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="storage-initializer"
Jul 06 21:45:14.265625 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265434    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="97118a8a-4346-4bd7-bac3-d3335301a659" containerName="main"
Jul 06 21:45:14.265625 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265446    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="llm-d-routing-sidecar"
Jul 06 21:45:14.265625 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.265453    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe05fbd-2e12-44fe-9d6a-cafee21ecb64" containerName="main"
Jul 06 21:45:14.270927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.270899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.274084 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.274063    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 06 21:45:14.281765 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.281737    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"]
Jul 06 21:45:14.289698 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.289668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:45:14.331045 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.331015    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:45:14.331288 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:14.331266    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:45:14.386485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.386437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.386655 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.386492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.386655 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.386536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.386778 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.386706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.386778 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.386742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.386875 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.386806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.386933 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.386887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9zh\" (UniqueName: \"kubernetes.io/projected/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kube-api-access-6c9zh\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.487727 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.487688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.487727 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.487727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.487758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.487801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.487823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.487846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488001 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.487899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9zh\" (UniqueName: \"kubernetes.io/projected/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kube-api-access-6c9zh\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488285 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.488237    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488330 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.488297    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488380 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.488313    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.488380 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.488340    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.489939 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.489920    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.490323 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.490304    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.497705 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.497680    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9zh\" (UniqueName: \"kubernetes.io/projected/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kube-api-access-6c9zh\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.583467 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.583386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:14.734254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:14.734225    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"]
Jul 06 21:45:14.734845 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:45:14.734820    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df37c3d_129e_4a51_ad14_6c1a6f5e9254.slice/crio-b95b6bf1967aee3a3ed1ba4c85cdd413b25f6c935ffb0169de90a4f1a91ec8ac WatchSource:0}: Error finding container b95b6bf1967aee3a3ed1ba4c85cdd413b25f6c935ffb0169de90a4f1a91ec8ac: Status 404 returned error can't find the container with id b95b6bf1967aee3a3ed1ba4c85cdd413b25f6c935ffb0169de90a4f1a91ec8ac
Jul 06 21:45:15.293911 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:15.293869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" event={"ID":"2df37c3d-129e-4a51-ad14-6c1a6f5e9254","Type":"ContainerStarted","Data":"29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8"}
Jul 06 21:45:15.294438 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:15.293943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" event={"ID":"2df37c3d-129e-4a51-ad14-6c1a6f5e9254","Type":"ContainerStarted","Data":"b95b6bf1967aee3a3ed1ba4c85cdd413b25f6c935ffb0169de90a4f1a91ec8ac"}
Jul 06 21:45:27.331616 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:27.331583    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:45:27.332144 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:27.331829    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:45:27.364661 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:27.364626    2582 generic.go:356] "Generic (PLEG): container finished" podID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerID="29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8" exitCode=0
Jul 06 21:45:27.364816 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:27.364665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" event={"ID":"2df37c3d-129e-4a51-ad14-6c1a6f5e9254","Type":"ContainerDied","Data":"29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8"}
Jul 06 21:45:28.373712 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:28.373669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" event={"ID":"2df37c3d-129e-4a51-ad14-6c1a6f5e9254","Type":"ContainerStarted","Data":"22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d"}
Jul 06 21:45:28.399186 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:28.399111    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podStartSLOduration=2.771467941 podStartE2EDuration="14.39909462s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.627626679s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:45:28.396193421 +0000 UTC m=+4225.730534037" watchObservedRunningTime="2026-07-06 21:45:28.39909462 +0000 UTC m=+4225.733435215"
Jul 06 21:45:28.584491 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:28.584447    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:34.584179 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:34.584140    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:45:34.585552 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:34.585525    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:45:39.331945 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:39.331904    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:45:39.332443 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:39.332173    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:45:44.584344 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:44.584304    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:45:52.331503 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:52.331468    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:45:52.331900 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:45:52.331729    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:45:54.584127 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:45:54.584085    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:46:03.335381 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:03.335349    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:46:03.335892 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:46:03.335568    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:46:04.584796 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:04.584752    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:46:14.584545 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:14.584499    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:46:17.331218 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:17.331185    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:46:17.332563 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:17.332523    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:46:17.332701 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:17.332618    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:46:17.332701 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:17.332663    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:46:17.665854 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:17.665757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"}
Jul 06 21:46:18.556653 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:18.556608    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:46:18.557938 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:18.557907    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:46:18.558108 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:18.557982    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:46:19.556666 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:19.556627    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:46:19.557113 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:19.556710    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:46:20.556402 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:20.556366    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:46:20.556585 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:20.556432    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:46:21.556377 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:21.556346    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:46:21.556774 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:21.556405    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:46:22.001086 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:22.001043    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:46:22.556761 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:22.556726    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:46:22.557171 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:22.556782    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:46:24.556215 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:24.556175    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 06 21:46:24.556657 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:24.556262    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:46:24.584378 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:24.584341    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:46:25.426070 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:25.426035    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:46:25.427305 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:25.427278    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:46:34.583836 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:34.583790    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:46:44.584435 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:44.584387    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:46:48.894981 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:48.894938    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904" exitCode=137
Jul 06 21:46:48.895501 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:48.894995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerDied","Data":"5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"}
Jul 06 21:46:48.895501 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:48.895125    2582 scope.go:117] "RemoveContainer" containerID="12cdce1034697c8bf895e76f66a4efe24769d99d3add4e8666ad269185585bb6"
Jul 06 21:46:48.895616 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:48.895570    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:46:48.895833 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:46:48.895803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:46:54.584627 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:54.584540    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:46:55.427207 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:55.427171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:46:55.427678 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:46:55.427660    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:46:55.427911 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:46:55.427888    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:47:04.584623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:04.584582    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:47:11.331443 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:11.331413    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:47:11.331871 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:47:11.331630    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:47:14.583949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:14.583903    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:47:24.593575 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:24.593540    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:47:24.601536 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:24.601511    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:47:25.337656 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:25.337627    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:47:25.337836 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:47:25.337810    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:47:37.331837 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:37.331803    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:47:37.332275 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:47:37.332033    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:47:52.330950 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:47:52.330916    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:47:52.331445 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:47:52.331212    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:48:06.331115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:48:06.331088    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:48:06.331503 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:48:06.331291    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:48:21.330791 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:48:21.330758    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:48:21.331302 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:48:21.331068    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:48:36.331814 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:48:36.331780    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:48:36.334463 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:48:36.332024    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:48:47.331535 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:48:47.331503    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:48:47.332011 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:48:47.331706    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:49:02.330800 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:49:02.330767    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:49:02.331500 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:49:02.330956    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:49:14.331273 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:49:14.331245    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:49:14.331771 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:49:14.331440    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:49:29.331672 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:49:29.331635    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:49:29.332166 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:49:29.331851    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:49:44.331068 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:49:44.331033    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:49:44.331576 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:49:44.331216    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:49:55.027772 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:49:55.027695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:49:55.330907 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:49:55.330805    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:49:55.331093 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:49:55.331045    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:50:03.732211 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:50:03.732184    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:50:03.745949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:50:03.745928    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:50:10.330857 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:50:10.330820    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:50:10.331338 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:50:10.331119    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:50:25.331662 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:50:25.331633    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:50:25.332174 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:50:25.331831    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:50:36.331675 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:50:36.331640    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:50:36.332163 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:50:36.331841    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:50:49.331779 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:50:49.331742    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:50:49.332285 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:50:49.332048    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:51:03.335023 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:03.334918    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:51:03.335616 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:51:03.335191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:51:14.331333 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:14.331303    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:51:14.331720 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:51:14.331504    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:51:26.331574 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:26.331497    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:51:26.331989 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:51:26.331688    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:51:37.331203 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:37.331170    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:51:37.331686 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:51:37.331431    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-vmnww_openshift-ingress(dfe3aa62-0935-48f7-a93d-8d21917d35a5)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5"
Jul 06 21:51:52.331207 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:52.331169    2582 scope.go:117] "RemoveContainer" containerID="5b40a9ce541df2334bcf9ecfefbe6ef89fbbf64257eb9ad801003ee3e3421904"
Jul 06 21:51:52.332465 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:52.332435    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:51:52.332539 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:52.332509    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:51:52.332578 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:52.332546    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:51:52.684843 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:52.684803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" event={"ID":"dfe3aa62-0935-48f7-a93d-8d21917d35a5","Type":"ContainerStarted","Data":"3df46880e6706fa46df965c7a318e74758c1fb259895c8a98e0af3813dbccf8b"}
Jul 06 21:51:53.555636 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:53.555601    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:51:53.556804 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:53.556786    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:51:53.556856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:53.556837    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:51:53.930103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:53.926796    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"]
Jul 06 21:51:53.954827 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:53.954797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:51:54.556393 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:54.556358    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:51:54.556788 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:54.556410    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:51:55.426941 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:55.426905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:51:55.556224 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:55.556189    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:51:55.556407 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:55.556244    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:51:56.556266 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:56.556226    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:51:56.556734 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:56.556289    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:51:57.556278 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:57.556245    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 06 21:51:57.556671 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:57.556296    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww" podUID="dfe3aa62-0935-48f7-a93d-8d21917d35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 06 21:51:59.318604 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:59.318567    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:51:59.320599 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:51:59.320570    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-vmnww"
Jul 06 21:52:08.952845 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:08.952804    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" containerID="cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402" gracePeriod=45
Jul 06 21:52:09.076085 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.076030    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 06 21:52:09.537249 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.537220    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:52:09.710733 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710699    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-home\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.710924 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4948f86-f0d1-4911-908f-7cb88db94468-tls-certs\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.710924 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710785    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-model-cache\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.710924 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-dshm\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.710924 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710863    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5dws\" (UniqueName: \"kubernetes.io/projected/d4948f86-f0d1-4911-908f-7cb88db94468-kube-api-access-n5dws\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.710924 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4948f86-f0d1-4911-908f-7cb88db94468-cabundle-cert\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.711211 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-kserve-provision-location\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.711211 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.710984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-tmp-dir\") pod \"d4948f86-f0d1-4911-908f-7cb88db94468\" (UID: \"d4948f86-f0d1-4911-908f-7cb88db94468\") "
Jul 06 21:52:09.711211 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.711115    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:52:09.711371 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.711136    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-home" (OuterVolumeSpecName: "home") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:52:09.711444 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.711426    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.711499 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.711448    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.711661 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.711639    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4948f86-f0d1-4911-908f-7cb88db94468-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:52:09.713077 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.713047    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4948f86-f0d1-4911-908f-7cb88db94468-kube-api-access-n5dws" (OuterVolumeSpecName: "kube-api-access-n5dws") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "kube-api-access-n5dws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:52:09.713431 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.713405    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-dshm" (OuterVolumeSpecName: "dshm") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:52:09.713550 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.713432    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4948f86-f0d1-4911-908f-7cb88db94468-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:52:09.722677 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.722653    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:52:09.761313 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.761270    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4948f86-f0d1-4911-908f-7cb88db94468" (UID: "d4948f86-f0d1-4911-908f-7cb88db94468"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:52:09.804292 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.804249    2582 generic.go:356] "Generic (PLEG): container finished" podID="d4948f86-f0d1-4911-908f-7cb88db94468" containerID="d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402" exitCode=0
Jul 06 21:52:09.804478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.804286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" event={"ID":"d4948f86-f0d1-4911-908f-7cb88db94468","Type":"ContainerDied","Data":"d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402"}
Jul 06 21:52:09.804478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.804323    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"
Jul 06 21:52:09.804478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.804381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t" event={"ID":"d4948f86-f0d1-4911-908f-7cb88db94468","Type":"ContainerDied","Data":"299d3e66b7e1cca6fc48a1997ee24266ef60d8c156d947ca88279100e74760b7"}
Jul 06 21:52:09.804609 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.804403    2582 scope.go:117] "RemoveContainer" containerID="d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402"
Jul 06 21:52:09.811842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.811822    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4948f86-f0d1-4911-908f-7cb88db94468-cabundle-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.811842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.811843    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.811999 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.811870    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.811999 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.811879    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4948f86-f0d1-4911-908f-7cb88db94468-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.811999 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.811887    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4948f86-f0d1-4911-908f-7cb88db94468-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.811999 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.811895    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5dws\" (UniqueName: \"kubernetes.io/projected/d4948f86-f0d1-4911-908f-7cb88db94468-kube-api-access-n5dws\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:52:09.814788 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.814772    2582 scope.go:117] "RemoveContainer" containerID="fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31"
Jul 06 21:52:09.823654 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.823641    2582 scope.go:117] "RemoveContainer" containerID="d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402"
Jul 06 21:52:09.823887 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:52:09.823867    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402\": container with ID starting with d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402 not found: ID does not exist" containerID="d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402"
Jul 06 21:52:09.823927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.823898    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402"} err="failed to get container status \"d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402\": rpc error: code = NotFound desc = could not find container \"d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402\": container with ID starting with d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402 not found: ID does not exist"
Jul 06 21:52:09.823927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.823923    2582 scope.go:117] "RemoveContainer" containerID="fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31"
Jul 06 21:52:09.824164 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:52:09.824142    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31\": container with ID starting with fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31 not found: ID does not exist" containerID="fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31"
Jul 06 21:52:09.824264 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.824168    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31"} err="failed to get container status \"fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31\": rpc error: code = NotFound desc = could not find container \"fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31\": container with ID starting with fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31 not found: ID does not exist"
Jul 06 21:52:09.829379 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.829359    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"]
Jul 06 21:52:09.835033 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:09.835012    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t"]
Jul 06 21:52:11.336908 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:11.336873    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" path="/var/lib/kubelet/pods/d4948f86-f0d1-4911-908f-7cb88db94468/volumes"
Jul 06 21:52:18.678526 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:18.678489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:52:36.374570 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.374537    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"]
Jul 06 21:52:36.374975 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.374949    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main"
Jul 06 21:52:36.375020 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.374980    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main"
Jul 06 21:52:36.375020 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.375005    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="storage-initializer"
Jul 06 21:52:36.375020 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.375011    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="storage-initializer"
Jul 06 21:52:36.375127 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.375089    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4948f86-f0d1-4911-908f-7cb88db94468" containerName="main"
Jul 06 21:52:36.378222 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.378206    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.381299 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.381274    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 06 21:52:36.389292 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.389265    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"]
Jul 06 21:52:36.430400 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.430351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:52:36.458166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.458134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lcg\" (UniqueName: \"kubernetes.io/projected/fba3a8bd-769b-45f5-86d4-2cc63c242029-kube-api-access-p8lcg\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.458166 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.458168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-tmp-dir\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.458374 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.458193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-kserve-provision-location\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.458374 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.458273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-home\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.458374 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.458297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-dshm\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.458374 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.458369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-model-cache\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.458532 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.458410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba3a8bd-769b-45f5-86d4-2cc63c242029-tls-certs\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.559614 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.559579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba3a8bd-769b-45f5-86d4-2cc63c242029-tls-certs\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.559797 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.559634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lcg\" (UniqueName: \"kubernetes.io/projected/fba3a8bd-769b-45f5-86d4-2cc63c242029-kube-api-access-p8lcg\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.559797 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.559655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-tmp-dir\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.559797 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.559704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-kserve-provision-location\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.559797 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.559759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-home\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.560046 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.559891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-dshm\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.560107 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.560067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-model-cache\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.560160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.560107    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-tmp-dir\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.560160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.560140    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-kserve-provision-location\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.560160 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.560154    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-home\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.560424 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.560402    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-model-cache\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.562129 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.562103    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-dshm\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.562331 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.562315    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba3a8bd-769b-45f5-86d4-2cc63c242029-tls-certs\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.568650 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.568627    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lcg\" (UniqueName: \"kubernetes.io/projected/fba3a8bd-769b-45f5-86d4-2cc63c242029-kube-api-access-p8lcg\") pod \"tls-verification-test-kserve-85684c76c-44jqz\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.688871 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.688835    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:36.716396 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.716367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:52:36.832943 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.832919    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"]
Jul 06 21:52:36.834660 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:52:36.834633    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba3a8bd_769b_45f5_86d4_2cc63c242029.slice/crio-81a035286a32809c9475b198a198a1e09c08bea9841ab8daa94bafbdbf465519 WatchSource:0}: Error finding container 81a035286a32809c9475b198a198a1e09c08bea9841ab8daa94bafbdbf465519: Status 404 returned error can't find the container with id 81a035286a32809c9475b198a198a1e09c08bea9841ab8daa94bafbdbf465519
Jul 06 21:52:36.838908 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.838887    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:52:36.958552 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.958469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" event={"ID":"fba3a8bd-769b-45f5-86d4-2cc63c242029","Type":"ContainerStarted","Data":"0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6"}
Jul 06 21:52:36.958552 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:36.958543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" event={"ID":"fba3a8bd-769b-45f5-86d4-2cc63c242029","Type":"ContainerStarted","Data":"81a035286a32809c9475b198a198a1e09c08bea9841ab8daa94bafbdbf465519"}
Jul 06 21:52:40.983118 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:40.983085    2582 generic.go:356] "Generic (PLEG): container finished" podID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerID="0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6" exitCode=0
Jul 06 21:52:40.983498 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:40.983162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" event={"ID":"fba3a8bd-769b-45f5-86d4-2cc63c242029","Type":"ContainerDied","Data":"0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6"}
Jul 06 21:52:41.991375 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:41.991330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" event={"ID":"fba3a8bd-769b-45f5-86d4-2cc63c242029","Type":"ContainerStarted","Data":"7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd"}
Jul 06 21:52:42.017463 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:42.017406    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podStartSLOduration=1.9944178830000001 podStartE2EDuration="6.017391781s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.022973898s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:52:42.015634482 +0000 UTC m=+4659.349975077" watchObservedRunningTime="2026-07-06 21:52:42.017391781 +0000 UTC m=+4659.351732376"
Jul 06 21:52:42.689206 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:42.689154    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:46.689551 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:46.689504    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:52:46.691156 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:46.691126    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:52:56.689689 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:52:56.689593    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:53:02.688941 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:02.688908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:53:06.690274 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:06.690226    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:53:16.689326 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:16.689280    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:53:26.690103 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:26.690053    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:53:29.277043 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:29.277000    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"]
Jul 06 21:53:29.322222 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:29.322092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:53:36.689671 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:36.689631    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:53:44.306123 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:44.306050    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" containerID="cri-o://22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d" gracePeriod=45
Jul 06 21:53:44.594649 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:44.594530    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 06 21:53:45.092558 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.092532    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:53:45.145027 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.144929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tmp-dir\") pod \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") "
Jul 06 21:53:45.145027 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.144996    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-model-cache\") pod \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") "
Jul 06 21:53:45.145261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145088    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-dshm\") pod \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") "
Jul 06 21:53:45.145261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145114    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-home\") pod \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") "
Jul 06 21:53:45.145261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c9zh\" (UniqueName: \"kubernetes.io/projected/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kube-api-access-6c9zh\") pod \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") "
Jul 06 21:53:45.145261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tls-certs\") pod \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") "
Jul 06 21:53:45.145261 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kserve-provision-location\") pod \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\" (UID: \"2df37c3d-129e-4a51-ad14-6c1a6f5e9254\") "
Jul 06 21:53:45.145525 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145321    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-model-cache" (OuterVolumeSpecName: "model-cache") pod "2df37c3d-129e-4a51-ad14-6c1a6f5e9254" (UID: "2df37c3d-129e-4a51-ad14-6c1a6f5e9254"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:53:45.145608 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145583    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:53:45.145945 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.145777    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-home" (OuterVolumeSpecName: "home") pod "2df37c3d-129e-4a51-ad14-6c1a6f5e9254" (UID: "2df37c3d-129e-4a51-ad14-6c1a6f5e9254"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:53:45.147783 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.147745    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2df37c3d-129e-4a51-ad14-6c1a6f5e9254" (UID: "2df37c3d-129e-4a51-ad14-6c1a6f5e9254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:53:45.148036 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.148009    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kube-api-access-6c9zh" (OuterVolumeSpecName: "kube-api-access-6c9zh") pod "2df37c3d-129e-4a51-ad14-6c1a6f5e9254" (UID: "2df37c3d-129e-4a51-ad14-6c1a6f5e9254"). InnerVolumeSpecName "kube-api-access-6c9zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:53:45.148118 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.148094    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-dshm" (OuterVolumeSpecName: "dshm") pod "2df37c3d-129e-4a51-ad14-6c1a6f5e9254" (UID: "2df37c3d-129e-4a51-ad14-6c1a6f5e9254"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:53:45.164291 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.164255    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2df37c3d-129e-4a51-ad14-6c1a6f5e9254" (UID: "2df37c3d-129e-4a51-ad14-6c1a6f5e9254"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:53:45.220912 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.220871    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2df37c3d-129e-4a51-ad14-6c1a6f5e9254" (UID: "2df37c3d-129e-4a51-ad14-6c1a6f5e9254"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:53:45.246301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.246265    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:53:45.246301 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.246294    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:53:45.246506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.246307    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9zh\" (UniqueName: \"kubernetes.io/projected/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kube-api-access-6c9zh\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:53:45.246506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.246322    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:53:45.246506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.246334    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:53:45.246506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.246365    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df37c3d-129e-4a51-ad14-6c1a6f5e9254-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:53:45.378908 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.378874    2582 generic.go:356] "Generic (PLEG): container finished" podID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerID="22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d" exitCode=0
Jul 06 21:53:45.379408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.378936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"
Jul 06 21:53:45.379408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.378936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" event={"ID":"2df37c3d-129e-4a51-ad14-6c1a6f5e9254","Type":"ContainerDied","Data":"22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d"}
Jul 06 21:53:45.379408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.379036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s" event={"ID":"2df37c3d-129e-4a51-ad14-6c1a6f5e9254","Type":"ContainerDied","Data":"b95b6bf1967aee3a3ed1ba4c85cdd413b25f6c935ffb0169de90a4f1a91ec8ac"}
Jul 06 21:53:45.379408 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.379060    2582 scope.go:117] "RemoveContainer" containerID="22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d"
Jul 06 21:53:45.388751 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.388734    2582 scope.go:117] "RemoveContainer" containerID="29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8"
Jul 06 21:53:45.397263 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.397203    2582 scope.go:117] "RemoveContainer" containerID="22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d"
Jul 06 21:53:45.397686 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:53:45.397664    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d\": container with ID starting with 22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d not found: ID does not exist" containerID="22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d"
Jul 06 21:53:45.397793 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.397694    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d"} err="failed to get container status \"22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d\": rpc error: code = NotFound desc = could not find container \"22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d\": container with ID starting with 22d7739adfe55b238c646f9dba315235a2f6127b5d15dcd69d192a4b94b9dc9d not found: ID does not exist"
Jul 06 21:53:45.397793 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.397721    2582 scope.go:117] "RemoveContainer" containerID="29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8"
Jul 06 21:53:45.398039 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:53:45.398016    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8\": container with ID starting with 29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8 not found: ID does not exist" containerID="29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8"
Jul 06 21:53:45.398099 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.398045    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8"} err="failed to get container status \"29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8\": rpc error: code = NotFound desc = could not find container \"29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8\": container with ID starting with 29848f097be99e801f63aa33d34069f30bfc638b05f7f9fbd2b4cf1a38fad7d8 not found: ID does not exist"
Jul 06 21:53:45.401176 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.401155    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"]
Jul 06 21:53:45.405214 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.405194    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s"]
Jul 06 21:53:45.821568 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:45.821536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:53:46.689749 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:46.689706    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:53:47.337681 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:47.337648    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" path="/var/lib/kubelet/pods/2df37c3d-129e-4a51-ad14-6c1a6f5e9254/volumes"
Jul 06 21:53:56.690279 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:53:56.690238    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:54:06.293051 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.293012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:06.413548 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.413512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:06.449817 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.449787    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:54:06.450209 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.450196    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main"
Jul 06 21:54:06.450209 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.450211    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main"
Jul 06 21:54:06.450292 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.450227    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="storage-initializer"
Jul 06 21:54:06.450292 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.450233    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="storage-initializer"
Jul 06 21:54:06.450356 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.450303    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df37c3d-129e-4a51-ad14-6c1a6f5e9254" containerName="main"
Jul 06 21:54:06.453224 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.453206    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.456005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.455957    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-s5558\""
Jul 06 21:54:06.456116 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.456059    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 06 21:54:06.465580 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.465557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:54:06.499235 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.499206    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 06 21:54:06.502105 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.502087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.504113 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.504086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:06.517120 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.517093    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 06 21:54:06.550300 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.550205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.550300 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.550262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85l9f\" (UniqueName: \"kubernetes.io/projected/536cd3e9-8a67-4d82-83f5-89265cae691d-kube-api-access-85l9f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.550495 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.550335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.550495 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.550383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.550495 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.550450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.550597 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.550538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536cd3e9-8a67-4d82-83f5-89265cae691d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.550597 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.550576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.651586 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.651758 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536cd3e9-8a67-4d82-83f5-89265cae691d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.651758 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.651837 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.651837 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85l9f\" (UniqueName: \"kubernetes.io/projected/536cd3e9-8a67-4d82-83f5-89265cae691d-kube-api-access-85l9f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.651916 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.651916 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvzc\" (UniqueName: \"kubernetes.io/projected/b23e2538-8b7c-4584-b346-07ac3a32931f-kube-api-access-8qvzc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.651916 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.652093 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.652093 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.651998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.652093 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.652093 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b23e2538-8b7c-4584-b346-07ac3a32931f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.652313 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652087    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.652313 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.652313 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652140    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.652313 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.652530 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652326    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.652530 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.652502    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.654095 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.654077    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.654542 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.654524    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536cd3e9-8a67-4d82-83f5-89265cae691d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.660031 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.660003    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85l9f\" (UniqueName: \"kubernetes.io/projected/536cd3e9-8a67-4d82-83f5-89265cae691d-kube-api-access-85l9f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.689900 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.689858    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:54:06.719268 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.719224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:06.753293 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvzc\" (UniqueName: \"kubernetes.io/projected/b23e2538-8b7c-4584-b346-07ac3a32931f-kube-api-access-8qvzc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753506 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b23e2538-8b7c-4584-b346-07ac3a32931f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753767 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753767 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753686    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.753973 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753938    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.754097 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.753989    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.754097 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.754033    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.755736 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.755708    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.756048 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.756028    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b23e2538-8b7c-4584-b346-07ac3a32931f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.761794 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.761767    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvzc\" (UniqueName: \"kubernetes.io/projected/b23e2538-8b7c-4584-b346-07ac3a32931f-kube-api-access-8qvzc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.764773 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.764702    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:06.813177 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.813090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:54:06.944472 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.944441    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:54:06.986642 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:06.986559    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 06 21:54:06.988176 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:54:06.988152    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23e2538_8b7c_4584_b346_07ac3a32931f.slice/crio-caffbb34ddc274f8591186425ac2873eca09580774d0eb19028b286051a75f5f WatchSource:0}: Error finding container caffbb34ddc274f8591186425ac2873eca09580774d0eb19028b286051a75f5f: Status 404 returned error can't find the container with id caffbb34ddc274f8591186425ac2873eca09580774d0eb19028b286051a75f5f
Jul 06 21:54:07.529779 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:07.529731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"536cd3e9-8a67-4d82-83f5-89265cae691d","Type":"ContainerStarted","Data":"5619e51a299b3249bcdd9905854e96e3801101de68337c8691d6d4f458f947eb"}
Jul 06 21:54:07.530291 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:07.529845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"536cd3e9-8a67-4d82-83f5-89265cae691d","Type":"ContainerStarted","Data":"c229862e802935da8256509203f1318b65b8a217ba743e3f4ac8cae56b738b62"}
Jul 06 21:54:07.532478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:07.532342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b23e2538-8b7c-4584-b346-07ac3a32931f","Type":"ContainerStarted","Data":"c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964"}
Jul 06 21:54:07.532478 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:07.532401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b23e2538-8b7c-4584-b346-07ac3a32931f","Type":"ContainerStarted","Data":"caffbb34ddc274f8591186425ac2873eca09580774d0eb19028b286051a75f5f"}
Jul 06 21:54:11.575208 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:11.575100    2582 generic.go:356] "Generic (PLEG): container finished" podID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerID="c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964" exitCode=0
Jul 06 21:54:11.575208 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:11.575184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b23e2538-8b7c-4584-b346-07ac3a32931f","Type":"ContainerDied","Data":"c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964"}
Jul 06 21:54:12.584382 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:12.584349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b23e2538-8b7c-4584-b346-07ac3a32931f","Type":"ContainerStarted","Data":"680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e"}
Jul 06 21:54:12.586075 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:12.586048    2582 generic.go:356] "Generic (PLEG): container finished" podID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerID="5619e51a299b3249bcdd9905854e96e3801101de68337c8691d6d4f458f947eb" exitCode=0
Jul 06 21:54:12.586189 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:12.586110    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"536cd3e9-8a67-4d82-83f5-89265cae691d","Type":"ContainerDied","Data":"5619e51a299b3249bcdd9905854e96e3801101de68337c8691d6d4f458f947eb"}
Jul 06 21:54:12.609020 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:12.608939    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.425988879 podStartE2EDuration="6.608917406s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.182928527s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:54:12.605325083 +0000 UTC m=+4749.939665679" watchObservedRunningTime="2026-07-06 21:54:12.608917406 +0000 UTC m=+4749.943258006"
Jul 06 21:54:13.597554 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:13.597514    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"536cd3e9-8a67-4d82-83f5-89265cae691d","Type":"ContainerStarted","Data":"fb140dcf3835279085adb8617a403c50ac680ddabe30058abbcbb5fd2bf33ce6"}
Jul 06 21:54:13.626181 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:13.626112    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.898963493 podStartE2EDuration="7.626088205s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.727124712s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:54:13.621257557 +0000 UTC m=+4750.955598178" watchObservedRunningTime="2026-07-06 21:54:13.626088205 +0000 UTC m=+4750.960428802"
Jul 06 21:54:13.765477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:13.765440    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:14.642126 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:14.642075    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:16.690218 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:16.690174    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:54:16.765668 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:16.765626    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:54:16.767694 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:16.767653    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:54:26.690074 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:26.689950    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Jul 06 21:54:26.766017 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:26.765949    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:54:36.701030 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:36.700994    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:54:36.712621 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:36.712587    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:54:36.765787 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:36.765745    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:54:41.961811 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:41.961772    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"]
Jul 06 21:54:41.971909 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:41.971867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:42.032840 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:42.032765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:54:46.766306 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:46.766255    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:54:56.765922 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:56.765881    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:54:56.999468 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:56.999421    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" containerID="cri-o://7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd" gracePeriod=45
Jul 06 21:54:57.772942 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.772913    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:54:57.917657 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.917565    2582 generic.go:356] "Generic (PLEG): container finished" podID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerID="7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd" exitCode=0
Jul 06 21:54:57.917657 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.917623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" event={"ID":"fba3a8bd-769b-45f5-86d4-2cc63c242029","Type":"ContainerDied","Data":"7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd"}
Jul 06 21:54:57.917657 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.917642    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"
Jul 06 21:54:57.917890 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.917709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" event={"ID":"fba3a8bd-769b-45f5-86d4-2cc63c242029","Type":"ContainerDied","Data":"81a035286a32809c9475b198a198a1e09c08bea9841ab8daa94bafbdbf465519"}
Jul 06 21:54:57.917890 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.917741    2582 scope.go:117] "RemoveContainer" containerID="7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd"
Jul 06 21:54:57.921237 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921208    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba3a8bd-769b-45f5-86d4-2cc63c242029-tls-certs\") pod \"fba3a8bd-769b-45f5-86d4-2cc63c242029\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") "
Jul 06 21:54:57.921379 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921254    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8lcg\" (UniqueName: \"kubernetes.io/projected/fba3a8bd-769b-45f5-86d4-2cc63c242029-kube-api-access-p8lcg\") pod \"fba3a8bd-769b-45f5-86d4-2cc63c242029\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") "
Jul 06 21:54:57.921379 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921344    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-model-cache\") pod \"fba3a8bd-769b-45f5-86d4-2cc63c242029\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") "
Jul 06 21:54:57.921379 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921375    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-tmp-dir\") pod \"fba3a8bd-769b-45f5-86d4-2cc63c242029\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") "
Jul 06 21:54:57.921555 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921539    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-home\") pod \"fba3a8bd-769b-45f5-86d4-2cc63c242029\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") "
Jul 06 21:54:57.921663 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921632    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-model-cache" (OuterVolumeSpecName: "model-cache") pod "fba3a8bd-769b-45f5-86d4-2cc63c242029" (UID: "fba3a8bd-769b-45f5-86d4-2cc63c242029"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:54:57.921748 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921671    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-dshm\") pod \"fba3a8bd-769b-45f5-86d4-2cc63c242029\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") "
Jul 06 21:54:57.921748 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.921704    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-kserve-provision-location\") pod \"fba3a8bd-769b-45f5-86d4-2cc63c242029\" (UID: \"fba3a8bd-769b-45f5-86d4-2cc63c242029\") "
Jul 06 21:54:57.922095 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.922063    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:54:57.922368 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.922250    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-home" (OuterVolumeSpecName: "home") pod "fba3a8bd-769b-45f5-86d4-2cc63c242029" (UID: "fba3a8bd-769b-45f5-86d4-2cc63c242029"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:54:57.924379 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.924326    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba3a8bd-769b-45f5-86d4-2cc63c242029-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fba3a8bd-769b-45f5-86d4-2cc63c242029" (UID: "fba3a8bd-769b-45f5-86d4-2cc63c242029"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:54:57.924653 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.924633    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-dshm" (OuterVolumeSpecName: "dshm") pod "fba3a8bd-769b-45f5-86d4-2cc63c242029" (UID: "fba3a8bd-769b-45f5-86d4-2cc63c242029"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:54:57.924778 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.924761    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba3a8bd-769b-45f5-86d4-2cc63c242029-kube-api-access-p8lcg" (OuterVolumeSpecName: "kube-api-access-p8lcg") pod "fba3a8bd-769b-45f5-86d4-2cc63c242029" (UID: "fba3a8bd-769b-45f5-86d4-2cc63c242029"). InnerVolumeSpecName "kube-api-access-p8lcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:54:57.929944 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.929916    2582 scope.go:117] "RemoveContainer" containerID="0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6"
Jul 06 21:54:57.936623 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.936595    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fba3a8bd-769b-45f5-86d4-2cc63c242029" (UID: "fba3a8bd-769b-45f5-86d4-2cc63c242029"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:54:57.954157 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.954134    2582 scope.go:117] "RemoveContainer" containerID="7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd"
Jul 06 21:54:57.954502 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:54:57.954477    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd\": container with ID starting with 7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd not found: ID does not exist" containerID="7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd"
Jul 06 21:54:57.954602 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.954514    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd"} err="failed to get container status \"7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd\": rpc error: code = NotFound desc = could not find container \"7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd\": container with ID starting with 7c1cad10e5d0cd96f0f7f1b190521796fbac49afbb80d53a76567fc42f6ab4dd not found: ID does not exist"
Jul 06 21:54:57.954602 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.954540    2582 scope.go:117] "RemoveContainer" containerID="0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6"
Jul 06 21:54:57.954839 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:54:57.954816    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6\": container with ID starting with 0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6 not found: ID does not exist" containerID="0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6"
Jul 06 21:54:57.954904 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:57.954854    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6"} err="failed to get container status \"0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6\": rpc error: code = NotFound desc = could not find container \"0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6\": container with ID starting with 0afba588135f0db798e5d379e9b7bc37bfadf9b52011ea0a568e17eaddac6cf6 not found: ID does not exist"
Jul 06 21:54:58.005983 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.005920    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fba3a8bd-769b-45f5-86d4-2cc63c242029" (UID: "fba3a8bd-769b-45f5-86d4-2cc63c242029"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:54:58.022931 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.022891    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:54:58.022931 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.022921    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:54:58.022931 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.022932    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:54:58.023262 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.022942    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba3a8bd-769b-45f5-86d4-2cc63c242029-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:54:58.023262 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.022953    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lcg\" (UniqueName: \"kubernetes.io/projected/fba3a8bd-769b-45f5-86d4-2cc63c242029-kube-api-access-p8lcg\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:54:58.023262 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.023005    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba3a8bd-769b-45f5-86d4-2cc63c242029-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:54:58.242764 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.242725    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"]
Jul 06 21:54:58.246866 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.246835    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz"]
Jul 06 21:54:58.701989 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:58.701929    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 06 21:54:59.338742 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:54:59.338700    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" path="/var/lib/kubelet/pods/fba3a8bd-769b-45f5-86d4-2cc63c242029/volumes"
Jul 06 21:55:03.775436 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:03.775411    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:55:03.792045 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:03.792020    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 21:55:06.765463 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:06.765413    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:55:16.765885 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:16.765842    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:55:26.765318 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:26.765272    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:55:36.765595 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:36.765547    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:55:45.480126 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.480091    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"]
Jul 06 21:55:45.480532 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.480519    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="storage-initializer"
Jul 06 21:55:45.480532 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.480533    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="storage-initializer"
Jul 06 21:55:45.480611 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.480553    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main"
Jul 06 21:55:45.480611 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.480559    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main"
Jul 06 21:55:45.480674 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.480626    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba3a8bd-769b-45f5-86d4-2cc63c242029" containerName="main"
Jul 06 21:55:45.483627 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.483610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.486309 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.486287    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 06 21:55:45.496984 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.496938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:55:45.501249 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.501222    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"]
Jul 06 21:55:45.604551 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.604510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqpw\" (UniqueName: \"kubernetes.io/projected/88681f6a-9771-4557-84cd-8d5e89839b7b-kube-api-access-6tqpw\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.604551 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.604554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-dshm\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.604792 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.604580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-kserve-provision-location\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.604792 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.604700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-home\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.604792 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.604751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681f6a-9771-4557-84cd-8d5e89839b7b-tls-certs\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.604902 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.604833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-model-cache\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.604902 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.604853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-tmp-dir\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.705824 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.705784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-dshm\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.705824 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.705823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-kserve-provision-location\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706059 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.705877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-home\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706059 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.705954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681f6a-9771-4557-84cd-8d5e89839b7b-tls-certs\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706153 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.706059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-model-cache\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706153 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.706084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-tmp-dir\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706153 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.706136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqpw\" (UniqueName: \"kubernetes.io/projected/88681f6a-9771-4557-84cd-8d5e89839b7b-kube-api-access-6tqpw\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706359 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.706334    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-kserve-provision-location\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706560 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.706394    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-home\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706664 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.706456    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-model-cache\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.706724 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.706669    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-tmp-dir\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.708194 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.708165    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-dshm\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.708525 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.708505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681f6a-9771-4557-84cd-8d5e89839b7b-tls-certs\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.716153 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.716124    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqpw\" (UniqueName: \"kubernetes.io/projected/88681f6a-9771-4557-84cd-8d5e89839b7b-kube-api-access-6tqpw\") pod \"prestop-hook-test-kserve-56bd76dbb7-ckl2g\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.805776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.805672    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:45.961925 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:45.961895    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"]
Jul 06 21:55:45.963714 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:55:45.963685    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88681f6a_9771_4557_84cd_8d5e89839b7b.slice/crio-ba99531eebc8a040dcbeaab7de69e4839237c203791815b8080ebd96b8371812 WatchSource:0}: Error finding container ba99531eebc8a040dcbeaab7de69e4839237c203791815b8080ebd96b8371812: Status 404 returned error can't find the container with id ba99531eebc8a040dcbeaab7de69e4839237c203791815b8080ebd96b8371812
Jul 06 21:55:46.226807 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:46.226761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" event={"ID":"88681f6a-9771-4557-84cd-8d5e89839b7b","Type":"ContainerStarted","Data":"c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68"}
Jul 06 21:55:46.227014 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:46.226847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" event={"ID":"88681f6a-9771-4557-84cd-8d5e89839b7b","Type":"ContainerStarted","Data":"ba99531eebc8a040dcbeaab7de69e4839237c203791815b8080ebd96b8371812"}
Jul 06 21:55:46.754458 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:46.754429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:55:46.765704 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:46.765658    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:55:50.261417 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:50.261379    2582 generic.go:356] "Generic (PLEG): container finished" podID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerID="c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68" exitCode=0
Jul 06 21:55:50.261915 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:50.261455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" event={"ID":"88681f6a-9771-4557-84cd-8d5e89839b7b","Type":"ContainerDied","Data":"c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68"}
Jul 06 21:55:51.271043 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:51.270999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" event={"ID":"88681f6a-9771-4557-84cd-8d5e89839b7b","Type":"ContainerStarted","Data":"40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643"}
Jul 06 21:55:51.297724 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:51.297661    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podStartSLOduration=2.736962804 podStartE2EDuration="6.297638335s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.560675531s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:55:51.293003666 +0000 UTC m=+4848.627344276" watchObservedRunningTime="2026-07-06 21:55:51.297638335 +0000 UTC m=+4848.631978932"
Jul 06 21:55:51.806873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:51.806830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:55.806440 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:55.806321    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:55:55.808046 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:55.808017    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:55:56.765632 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:55:56.765575    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:56:05.806556 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:05.806501    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:56:06.766045 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:06.765991    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:56:15.806705 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:15.806648    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:56:16.766011 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:16.765946    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:56:19.625724 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:19.625690    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:56:25.806246 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:25.806199    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:56:26.766185 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:26.766127    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:56:35.806986 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:35.806926    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:56:36.774850 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:36.774820    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:56:36.783520 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:36.783492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:56:45.806079 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:45.806030    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:56:54.972295 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:54.972263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:56:55.073598 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:55.073567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:56:55.806133 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:56:55.806085    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:57:05.806501 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:05.806461    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:57:15.806688 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:15.806640    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:57:25.806326 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:25.806223    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:57:35.806856 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:35.806809    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Jul 06 21:57:45.817005 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:45.816957    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:57:45.825355 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:45.825326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:57:47.728284 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.728246    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"]
Jul 06 21:57:47.751946 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.751916    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"]
Jul 06 21:57:47.757275 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.757249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.768594 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.768564    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"]
Jul 06 21:57:47.846944 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.846911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crts4\" (UniqueName: \"kubernetes.io/projected/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kube-api-access-crts4\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.846944 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.846949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tls-certs\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.847217 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.847017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kserve-provision-location\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.847217 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.847094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-home\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.847217 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.847108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-dshm\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.847217 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.847159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-model-cache\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.847370 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.847258    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tmp-dir\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.948547 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.948498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crts4\" (UniqueName: \"kubernetes.io/projected/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kube-api-access-crts4\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.948547 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.948540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tls-certs\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.948842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.948582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kserve-provision-location\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.948842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.948640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-home\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.948842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.948654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-dshm\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.948842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.948668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-model-cache\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.948842 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.948701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tmp-dir\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.949172 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.949148    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-home\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.949232 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.949212    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kserve-provision-location\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.949308 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.949280    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tmp-dir\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.949458 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.949313    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-model-cache\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.951122 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.951091    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-dshm\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.951364 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.951348    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tls-certs\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:47.956948 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:47.956924    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crts4\" (UniqueName: \"kubernetes.io/projected/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kube-api-access-crts4\") pod \"prestop-hook-test-kserve-56bd76dbb7-sgkcg\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:48.069695 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:48.069603    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:48.216559 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:48.216530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"]
Jul 06 21:57:48.217455 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:57:48.217425    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60035eb5_7f1e_4d98_acb5_1869f637f4e4.slice/crio-0a60efcd0e0dee670821e456eb39b457a02047196fc4fbb69cb5e5ec40563061 WatchSource:0}: Error finding container 0a60efcd0e0dee670821e456eb39b457a02047196fc4fbb69cb5e5ec40563061: Status 404 returned error can't find the container with id 0a60efcd0e0dee670821e456eb39b457a02047196fc4fbb69cb5e5ec40563061
Jul 06 21:57:49.017106 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:49.017069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" event={"ID":"60035eb5-7f1e-4d98-acb5-1869f637f4e4","Type":"ContainerStarted","Data":"4b2dbd22cb1f0d714686b6e0b70325d1369ddff019f3bb79ad49a63ab053a21b"}
Jul 06 21:57:49.017529 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:49.017134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" event={"ID":"60035eb5-7f1e-4d98-acb5-1869f637f4e4","Type":"ContainerStarted","Data":"0a60efcd0e0dee670821e456eb39b457a02047196fc4fbb69cb5e5ec40563061"}
Jul 06 21:57:53.046314 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:53.046221    2582 generic.go:356] "Generic (PLEG): container finished" podID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerID="4b2dbd22cb1f0d714686b6e0b70325d1369ddff019f3bb79ad49a63ab053a21b" exitCode=0
Jul 06 21:57:53.046314 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:53.046268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" event={"ID":"60035eb5-7f1e-4d98-acb5-1869f637f4e4","Type":"ContainerDied","Data":"4b2dbd22cb1f0d714686b6e0b70325d1369ddff019f3bb79ad49a63ab053a21b"}
Jul 06 21:57:54.054782 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:54.054744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" event={"ID":"60035eb5-7f1e-4d98-acb5-1869f637f4e4","Type":"ContainerStarted","Data":"437e273802fa35242806ee3ed9ba80b85be1c5fb5a97a6701fb89d93f48d3e1e"}
Jul 06 21:57:54.070605 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:54.070570    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:54.086044 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:54.085986    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" podStartSLOduration=2.597946109 podStartE2EDuration="7.085943514s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.487997405s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:57:54.079563451 +0000 UTC m=+4971.413904046" watchObservedRunningTime="2026-07-06 21:57:54.085943514 +0000 UTC m=+4971.420284114"
Jul 06 21:57:58.069898 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:58.069866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:57:58.071572 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:57:58.071544    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Jul 06 21:58:02.758705 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:02.758663    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main" containerID="cri-o://40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643" gracePeriod=45
Jul 06 21:58:03.556314 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.556243    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:58:03.711873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.711837    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-kserve-provision-location\") pod \"88681f6a-9771-4557-84cd-8d5e89839b7b\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") "
Jul 06 21:58:03.712168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.711903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tqpw\" (UniqueName: \"kubernetes.io/projected/88681f6a-9771-4557-84cd-8d5e89839b7b-kube-api-access-6tqpw\") pod \"88681f6a-9771-4557-84cd-8d5e89839b7b\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") "
Jul 06 21:58:03.712168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.711946    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-model-cache\") pod \"88681f6a-9771-4557-84cd-8d5e89839b7b\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") "
Jul 06 21:58:03.712168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.712008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-tmp-dir\") pod \"88681f6a-9771-4557-84cd-8d5e89839b7b\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") "
Jul 06 21:58:03.712168 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.712162    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-home\") pod \"88681f6a-9771-4557-84cd-8d5e89839b7b\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") "
Jul 06 21:58:03.712400 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.712202    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-dshm\") pod \"88681f6a-9771-4557-84cd-8d5e89839b7b\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") "
Jul 06 21:58:03.712400 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.712199    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "88681f6a-9771-4557-84cd-8d5e89839b7b" (UID: "88681f6a-9771-4557-84cd-8d5e89839b7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:03.712400 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.712251    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681f6a-9771-4557-84cd-8d5e89839b7b-tls-certs\") pod \"88681f6a-9771-4557-84cd-8d5e89839b7b\" (UID: \"88681f6a-9771-4557-84cd-8d5e89839b7b\") "
Jul 06 21:58:03.712613 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.712590    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:03.712743 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.712717    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-home" (OuterVolumeSpecName: "home") pod "88681f6a-9771-4557-84cd-8d5e89839b7b" (UID: "88681f6a-9771-4557-84cd-8d5e89839b7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:03.714872 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.714839    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88681f6a-9771-4557-84cd-8d5e89839b7b-kube-api-access-6tqpw" (OuterVolumeSpecName: "kube-api-access-6tqpw") pod "88681f6a-9771-4557-84cd-8d5e89839b7b" (UID: "88681f6a-9771-4557-84cd-8d5e89839b7b"). InnerVolumeSpecName "kube-api-access-6tqpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:58:03.715022 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.714869    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88681f6a-9771-4557-84cd-8d5e89839b7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88681f6a-9771-4557-84cd-8d5e89839b7b" (UID: "88681f6a-9771-4557-84cd-8d5e89839b7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:58:03.715022 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.714877    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-dshm" (OuterVolumeSpecName: "dshm") pod "88681f6a-9771-4557-84cd-8d5e89839b7b" (UID: "88681f6a-9771-4557-84cd-8d5e89839b7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:03.726755 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.726718    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88681f6a-9771-4557-84cd-8d5e89839b7b" (UID: "88681f6a-9771-4557-84cd-8d5e89839b7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:03.768275 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.768230    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88681f6a-9771-4557-84cd-8d5e89839b7b" (UID: "88681f6a-9771-4557-84cd-8d5e89839b7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:03.813949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.813847    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:03.813949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.813878    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88681f6a-9771-4557-84cd-8d5e89839b7b-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:03.813949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.813891    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:03.813949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.813900    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tqpw\" (UniqueName: \"kubernetes.io/projected/88681f6a-9771-4557-84cd-8d5e89839b7b-kube-api-access-6tqpw\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:03.813949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.813909    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:03.813949 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:03.813918    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88681f6a-9771-4557-84cd-8d5e89839b7b-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:04.120956 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.120880    2582 generic.go:356] "Generic (PLEG): container finished" podID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerID="40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643" exitCode=0
Jul 06 21:58:04.121121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.120980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" event={"ID":"88681f6a-9771-4557-84cd-8d5e89839b7b","Type":"ContainerDied","Data":"40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643"}
Jul 06 21:58:04.121121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.121063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g" event={"ID":"88681f6a-9771-4557-84cd-8d5e89839b7b","Type":"ContainerDied","Data":"ba99531eebc8a040dcbeaab7de69e4839237c203791815b8080ebd96b8371812"}
Jul 06 21:58:04.121121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.120988    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"
Jul 06 21:58:04.121121 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.121087    2582 scope.go:117] "RemoveContainer" containerID="40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643"
Jul 06 21:58:04.131576 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.131555    2582 scope.go:117] "RemoveContainer" containerID="c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68"
Jul 06 21:58:04.140485 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.140466    2582 scope.go:117] "RemoveContainer" containerID="40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643"
Jul 06 21:58:04.140748 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:58:04.140725    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643\": container with ID starting with 40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643 not found: ID does not exist" containerID="40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643"
Jul 06 21:58:04.140810 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.140757    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643"} err="failed to get container status \"40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643\": rpc error: code = NotFound desc = could not find container \"40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643\": container with ID starting with 40920e59d03e5488f64b045cbd5ed7aa24a57665afbf720ae7819906e11d6643 not found: ID does not exist"
Jul 06 21:58:04.140810 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.140774    2582 scope.go:117] "RemoveContainer" containerID="c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68"
Jul 06 21:58:04.141057 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:58:04.141034    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68\": container with ID starting with c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68 not found: ID does not exist" containerID="c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68"
Jul 06 21:58:04.141105 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.141068    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68"} err="failed to get container status \"c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68\": rpc error: code = NotFound desc = could not find container \"c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68\": container with ID starting with c39c00e613abbd784c32cd3cb830109808a0a8b810e47861a69e02d8717b1f68 not found: ID does not exist"
Jul 06 21:58:04.148635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.148609    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"]
Jul 06 21:58:04.154204 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:04.154181    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g"]
Jul 06 21:58:05.337305 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:05.337270    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" path="/var/lib/kubelet/pods/88681f6a-9771-4557-84cd-8d5e89839b7b/volumes"
Jul 06 21:58:06.133154 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:06.133118    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"]
Jul 06 21:58:06.142671 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:06.142642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:06.242861 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:06.242824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:12.607409 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:12.607374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:12.633074 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:12.633040    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:58:12.748140 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:12.748068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:21.161210 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:21.161170    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerName="main" containerID="cri-o://437e273802fa35242806ee3ed9ba80b85be1c5fb5a97a6701fb89d93f48d3e1e" gracePeriod=45
Jul 06 21:58:27.657700 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:27.657659    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" containerID="cri-o://fb140dcf3835279085adb8617a403c50ac680ddabe30058abbcbb5fd2bf33ce6" gracePeriod=45
Jul 06 21:58:27.776401 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:27.776359    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 06 21:58:28.274712 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.274667    2582 generic.go:356] "Generic (PLEG): container finished" podID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerID="fb140dcf3835279085adb8617a403c50ac680ddabe30058abbcbb5fd2bf33ce6" exitCode=0
Jul 06 21:58:28.274851 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.274732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"536cd3e9-8a67-4d82-83f5-89265cae691d","Type":"ContainerDied","Data":"fb140dcf3835279085adb8617a403c50ac680ddabe30058abbcbb5fd2bf33ce6"}
Jul 06 21:58:28.420176 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.420145    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:58:28.452422 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452345    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-tmp-dir\") pod \"536cd3e9-8a67-4d82-83f5-89265cae691d\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") "
Jul 06 21:58:28.452579 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-dshm\") pod \"536cd3e9-8a67-4d82-83f5-89265cae691d\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") "
Jul 06 21:58:28.452579 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-model-cache\") pod \"536cd3e9-8a67-4d82-83f5-89265cae691d\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") "
Jul 06 21:58:28.452579 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-home\") pod \"536cd3e9-8a67-4d82-83f5-89265cae691d\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") "
Jul 06 21:58:28.452579 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-kserve-provision-location\") pod \"536cd3e9-8a67-4d82-83f5-89265cae691d\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") "
Jul 06 21:58:28.452873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452602    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85l9f\" (UniqueName: \"kubernetes.io/projected/536cd3e9-8a67-4d82-83f5-89265cae691d-kube-api-access-85l9f\") pod \"536cd3e9-8a67-4d82-83f5-89265cae691d\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") "
Jul 06 21:58:28.452873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536cd3e9-8a67-4d82-83f5-89265cae691d-tls-certs\") pod \"536cd3e9-8a67-4d82-83f5-89265cae691d\" (UID: \"536cd3e9-8a67-4d82-83f5-89265cae691d\") "
Jul 06 21:58:28.452873 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.452801    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-model-cache" (OuterVolumeSpecName: "model-cache") pod "536cd3e9-8a67-4d82-83f5-89265cae691d" (UID: "536cd3e9-8a67-4d82-83f5-89265cae691d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:28.453085 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.453065    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:28.453202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.453177    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-home" (OuterVolumeSpecName: "home") pod "536cd3e9-8a67-4d82-83f5-89265cae691d" (UID: "536cd3e9-8a67-4d82-83f5-89265cae691d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:28.455940 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.455393    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536cd3e9-8a67-4d82-83f5-89265cae691d-kube-api-access-85l9f" (OuterVolumeSpecName: "kube-api-access-85l9f") pod "536cd3e9-8a67-4d82-83f5-89265cae691d" (UID: "536cd3e9-8a67-4d82-83f5-89265cae691d"). InnerVolumeSpecName "kube-api-access-85l9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:58:28.456423 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.456390    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-dshm" (OuterVolumeSpecName: "dshm") pod "536cd3e9-8a67-4d82-83f5-89265cae691d" (UID: "536cd3e9-8a67-4d82-83f5-89265cae691d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:28.456565 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.456423    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536cd3e9-8a67-4d82-83f5-89265cae691d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "536cd3e9-8a67-4d82-83f5-89265cae691d" (UID: "536cd3e9-8a67-4d82-83f5-89265cae691d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:58:28.466813 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.466788    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "536cd3e9-8a67-4d82-83f5-89265cae691d" (UID: "536cd3e9-8a67-4d82-83f5-89265cae691d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:28.510115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.510070    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "536cd3e9-8a67-4d82-83f5-89265cae691d" (UID: "536cd3e9-8a67-4d82-83f5-89265cae691d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:28.554250 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.554220    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/536cd3e9-8a67-4d82-83f5-89265cae691d-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:28.554250 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.554247    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:28.554250 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.554255    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:28.554464 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.554264    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:28.554464 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.554273    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536cd3e9-8a67-4d82-83f5-89265cae691d-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:28.554464 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:28.554282    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85l9f\" (UniqueName: \"kubernetes.io/projected/536cd3e9-8a67-4d82-83f5-89265cae691d-kube-api-access-85l9f\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:29.283488 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.283444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"536cd3e9-8a67-4d82-83f5-89265cae691d","Type":"ContainerDied","Data":"c229862e802935da8256509203f1318b65b8a217ba743e3f4ac8cae56b738b62"}
Jul 06 21:58:29.283488 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.283478    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 06 21:58:29.284055 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.283508    2582 scope.go:117] "RemoveContainer" containerID="fb140dcf3835279085adb8617a403c50ac680ddabe30058abbcbb5fd2bf33ce6"
Jul 06 21:58:29.293307 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.293289    2582 scope.go:117] "RemoveContainer" containerID="5619e51a299b3249bcdd9905854e96e3801101de68337c8691d6d4f458f947eb"
Jul 06 21:58:29.311010 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.310982    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:58:29.315208 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.315185    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 06 21:58:29.337780 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.337750    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" path="/var/lib/kubelet/pods/536cd3e9-8a67-4d82-83f5-89265cae691d/volumes"
Jul 06 21:58:29.338385 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:29.338366    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 06 21:58:44.361511 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:44.361471    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerName="main" containerID="cri-o://680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e" gracePeriod=45
Jul 06 21:58:45.123420 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.123393    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:58:45.217477 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.217445    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qvzc\" (UniqueName: \"kubernetes.io/projected/b23e2538-8b7c-4584-b346-07ac3a32931f-kube-api-access-8qvzc\") pod \"b23e2538-8b7c-4584-b346-07ac3a32931f\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") "
Jul 06 21:58:45.217672 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.217519    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-kserve-provision-location\") pod \"b23e2538-8b7c-4584-b346-07ac3a32931f\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") "
Jul 06 21:58:45.217672 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.217544    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-home\") pod \"b23e2538-8b7c-4584-b346-07ac3a32931f\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") "
Jul 06 21:58:45.217672 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.217586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-tmp-dir\") pod \"b23e2538-8b7c-4584-b346-07ac3a32931f\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") "
Jul 06 21:58:45.217672 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.217635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b23e2538-8b7c-4584-b346-07ac3a32931f-tls-certs\") pod \"b23e2538-8b7c-4584-b346-07ac3a32931f\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") "
Jul 06 21:58:45.217672 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.217655    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-model-cache\") pod \"b23e2538-8b7c-4584-b346-07ac3a32931f\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") "
Jul 06 21:58:45.217976 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.217689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-dshm\") pod \"b23e2538-8b7c-4584-b346-07ac3a32931f\" (UID: \"b23e2538-8b7c-4584-b346-07ac3a32931f\") "
Jul 06 21:58:45.218300 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.218274    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-home" (OuterVolumeSpecName: "home") pod "b23e2538-8b7c-4584-b346-07ac3a32931f" (UID: "b23e2538-8b7c-4584-b346-07ac3a32931f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:45.218599 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.218574    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b23e2538-8b7c-4584-b346-07ac3a32931f" (UID: "b23e2538-8b7c-4584-b346-07ac3a32931f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:45.220266 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.220233    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23e2538-8b7c-4584-b346-07ac3a32931f-kube-api-access-8qvzc" (OuterVolumeSpecName: "kube-api-access-8qvzc") pod "b23e2538-8b7c-4584-b346-07ac3a32931f" (UID: "b23e2538-8b7c-4584-b346-07ac3a32931f"). InnerVolumeSpecName "kube-api-access-8qvzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:58:45.220374 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.220306    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-dshm" (OuterVolumeSpecName: "dshm") pod "b23e2538-8b7c-4584-b346-07ac3a32931f" (UID: "b23e2538-8b7c-4584-b346-07ac3a32931f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:45.222228 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.222203    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23e2538-8b7c-4584-b346-07ac3a32931f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b23e2538-8b7c-4584-b346-07ac3a32931f" (UID: "b23e2538-8b7c-4584-b346-07ac3a32931f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:58:45.235142 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.235113    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b23e2538-8b7c-4584-b346-07ac3a32931f" (UID: "b23e2538-8b7c-4584-b346-07ac3a32931f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:45.263894 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.263859    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b23e2538-8b7c-4584-b346-07ac3a32931f" (UID: "b23e2538-8b7c-4584-b346-07ac3a32931f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:45.319439 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.319398    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:45.319439 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.319439    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:45.319635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.319455    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:45.319635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.319469    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b23e2538-8b7c-4584-b346-07ac3a32931f-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:45.319635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.319482    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:45.319635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.319493    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b23e2538-8b7c-4584-b346-07ac3a32931f-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:45.319635 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.319505    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvzc\" (UniqueName: \"kubernetes.io/projected/b23e2538-8b7c-4584-b346-07ac3a32931f-kube-api-access-8qvzc\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:58:45.384730 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.384693    2582 generic.go:356] "Generic (PLEG): container finished" podID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerID="680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e" exitCode=0
Jul 06 21:58:45.385216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.384749    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 06 21:58:45.385216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.384786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b23e2538-8b7c-4584-b346-07ac3a32931f","Type":"ContainerDied","Data":"680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e"}
Jul 06 21:58:45.385216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.384857    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b23e2538-8b7c-4584-b346-07ac3a32931f","Type":"ContainerDied","Data":"caffbb34ddc274f8591186425ac2873eca09580774d0eb19028b286051a75f5f"}
Jul 06 21:58:45.385216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.384894    2582 scope.go:117] "RemoveContainer" containerID="680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e"
Jul 06 21:58:45.393997 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.393956    2582 scope.go:117] "RemoveContainer" containerID="c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964"
Jul 06 21:58:45.404799 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.404773    2582 scope.go:117] "RemoveContainer" containerID="680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e"
Jul 06 21:58:45.405117 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:58:45.405094    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e\": container with ID starting with 680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e not found: ID does not exist" containerID="680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e"
Jul 06 21:58:45.405202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.405124    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e"} err="failed to get container status \"680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e\": rpc error: code = NotFound desc = could not find container \"680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e\": container with ID starting with 680d532bcf35365171c13645a12d6f85b9ad020c30f996caeb4a6956d5e6346e not found: ID does not exist"
Jul 06 21:58:45.405202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.405142    2582 scope.go:117] "RemoveContainer" containerID="c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964"
Jul 06 21:58:45.405417 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:58:45.405391    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964\": container with ID starting with c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964 not found: ID does not exist" containerID="c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964"
Jul 06 21:58:45.405531 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.405425    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964"} err="failed to get container status \"c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964\": rpc error: code = NotFound desc = could not find container \"c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964\": container with ID starting with c1d93d3306f9f1882088bc8688d71a2dfad8cc5f41c171fc836086077e65c964 not found: ID does not exist"
Jul 06 21:58:45.406801 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.406779    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 06 21:58:45.411090 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.411068    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 06 21:58:45.857737 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:45.857699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:58:47.338288 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:58:47.338243    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b23e2538-8b7c-4584-b346-07ac3a32931f" path="/var/lib/kubelet/pods/b23e2538-8b7c-4584-b346-07ac3a32931f/volumes"
Jul 06 21:59:01.556589 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.556500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:01.796218 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796180    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"]
Jul 06 21:59:01.796685 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796654    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="storage-initializer"
Jul 06 21:59:01.796685 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796675    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="storage-initializer"
Jul 06 21:59:01.796685 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796693    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerName="storage-initializer"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796699    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerName="storage-initializer"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796712    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796717    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796723    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796728    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796734    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796739    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796760    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="storage-initializer"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796769    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="storage-initializer"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796848    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88681f6a-9771-4557-84cd-8d5e89839b7b" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796858    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="536cd3e9-8a67-4d82-83f5-89265cae691d" containerName="main"
Jul 06 21:59:01.796951 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.796865    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b23e2538-8b7c-4584-b346-07ac3a32931f" containerName="main"
Jul 06 21:59:01.799115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.799093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.803296 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.803094    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-xtgwr\""
Jul 06 21:59:01.803296 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.803154    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 06 21:59:01.812152 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.812128    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"]
Jul 06 21:59:01.829813 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.829777    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:01.880603 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.880570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcxv\" (UniqueName: \"kubernetes.io/projected/a311dd91-c18b-40a3-8942-ab9407d33d99-kube-api-access-zhcxv\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.880769 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.880611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a311dd91-c18b-40a3-8942-ab9407d33d99-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.880769 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.880687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a311dd91-c18b-40a3-8942-ab9407d33d99-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.981712 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.981671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a311dd91-c18b-40a3-8942-ab9407d33d99-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.981901 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.981773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcxv\" (UniqueName: \"kubernetes.io/projected/a311dd91-c18b-40a3-8942-ab9407d33d99-kube-api-access-zhcxv\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.981901 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.981809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a311dd91-c18b-40a3-8942-ab9407d33d99-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.982254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.982230    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a311dd91-c18b-40a3-8942-ab9407d33d99-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.984271 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.984250    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a311dd91-c18b-40a3-8942-ab9407d33d99-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:01.990808 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:01.990778    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcxv\" (UniqueName: \"kubernetes.io/projected/a311dd91-c18b-40a3-8942-ab9407d33d99-kube-api-access-zhcxv\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:02.111487 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:02.111390    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:02.264898 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:02.264871    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"]
Jul 06 21:59:02.267279 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:59:02.267253    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda311dd91_c18b_40a3_8942_ab9407d33d99.slice/crio-15db0a5f0283b5f71cbf966edd46bc25cdc9ab5748d0476a38d749c3e2d1ad79 WatchSource:0}: Error finding container 15db0a5f0283b5f71cbf966edd46bc25cdc9ab5748d0476a38d749c3e2d1ad79: Status 404 returned error can't find the container with id 15db0a5f0283b5f71cbf966edd46bc25cdc9ab5748d0476a38d749c3e2d1ad79
Jul 06 21:59:02.490240 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:02.490206    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" event={"ID":"a311dd91-c18b-40a3-8942-ab9407d33d99","Type":"ContainerStarted","Data":"0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686"}
Jul 06 21:59:02.490428 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:02.490262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" event={"ID":"a311dd91-c18b-40a3-8942-ab9407d33d99","Type":"ContainerStarted","Data":"15db0a5f0283b5f71cbf966edd46bc25cdc9ab5748d0476a38d749c3e2d1ad79"}
Jul 06 21:59:02.490428 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:02.490420    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:06.526019 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.525986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-56bd76dbb7-sgkcg_60035eb5-7f1e-4d98-acb5-1869f637f4e4/main/0.log"
Jul 06 21:59:06.526486 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.526403    2582 generic.go:356] "Generic (PLEG): container finished" podID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerID="437e273802fa35242806ee3ed9ba80b85be1c5fb5a97a6701fb89d93f48d3e1e" exitCode=137
Jul 06 21:59:06.526486 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.526462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" event={"ID":"60035eb5-7f1e-4d98-acb5-1869f637f4e4","Type":"ContainerDied","Data":"437e273802fa35242806ee3ed9ba80b85be1c5fb5a97a6701fb89d93f48d3e1e"}
Jul 06 21:59:06.700411 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.700342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:06.866935 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.866911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-56bd76dbb7-sgkcg_60035eb5-7f1e-4d98-acb5-1869f637f4e4/main/0.log"
Jul 06 21:59:06.868590 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.868574    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:59:06.902984 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.902908    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" podStartSLOduration=5.902883266 podStartE2EDuration="5.902883266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:59:02.522236764 +0000 UTC m=+5039.856577380" watchObservedRunningTime="2026-07-06 21:59:06.902883266 +0000 UTC m=+5044.237223862"
Jul 06 21:59:06.936560 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.936535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kserve-provision-location\") pod \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") "
Jul 06 21:59:06.936809 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.936794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tmp-dir\") pod \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") "
Jul 06 21:59:06.937170 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.937155    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-model-cache\") pod \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") "
Jul 06 21:59:06.937482 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.937472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tls-certs\") pod \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") "
Jul 06 21:59:06.937815 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.937803    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-dshm\") pod \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") "
Jul 06 21:59:06.938149 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.938135    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crts4\" (UniqueName: \"kubernetes.io/projected/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kube-api-access-crts4\") pod \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") "
Jul 06 21:59:06.938332 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.938322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-home\") pod \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\" (UID: \"60035eb5-7f1e-4d98-acb5-1869f637f4e4\") "
Jul 06 21:59:06.938913 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.937111    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60035eb5-7f1e-4d98-acb5-1869f637f4e4" (UID: "60035eb5-7f1e-4d98-acb5-1869f637f4e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:06.938913 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.937438    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "60035eb5-7f1e-4d98-acb5-1869f637f4e4" (UID: "60035eb5-7f1e-4d98-acb5-1869f637f4e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:06.938913 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.938763    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-home" (OuterVolumeSpecName: "home") pod "60035eb5-7f1e-4d98-acb5-1869f637f4e4" (UID: "60035eb5-7f1e-4d98-acb5-1869f637f4e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:06.939494 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.939475    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:06.939776 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.939760    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:06.939927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.939914    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:06.940052 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.939650    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60035eb5-7f1e-4d98-acb5-1869f637f4e4" (UID: "60035eb5-7f1e-4d98-acb5-1869f637f4e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:59:06.940161 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.939712    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-dshm" (OuterVolumeSpecName: "dshm") pod "60035eb5-7f1e-4d98-acb5-1869f637f4e4" (UID: "60035eb5-7f1e-4d98-acb5-1869f637f4e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:06.940514 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:06.940493    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kube-api-access-crts4" (OuterVolumeSpecName: "kube-api-access-crts4") pod "60035eb5-7f1e-4d98-acb5-1869f637f4e4" (UID: "60035eb5-7f1e-4d98-acb5-1869f637f4e4"). InnerVolumeSpecName "kube-api-access-crts4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:59:07.024751 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.024711    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60035eb5-7f1e-4d98-acb5-1869f637f4e4" (UID: "60035eb5-7f1e-4d98-acb5-1869f637f4e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:07.041432 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.041391    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60035eb5-7f1e-4d98-acb5-1869f637f4e4-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:07.041432 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.041424    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:07.041634 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.041438    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crts4\" (UniqueName: \"kubernetes.io/projected/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kube-api-access-crts4\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:07.041634 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.041451    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60035eb5-7f1e-4d98-acb5-1869f637f4e4-kserve-provision-location\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:07.533953 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.533924    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-56bd76dbb7-sgkcg_60035eb5-7f1e-4d98-acb5-1869f637f4e4/main/0.log"
Jul 06 21:59:07.534445 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.534426    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"
Jul 06 21:59:07.534520 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.534500    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg" event={"ID":"60035eb5-7f1e-4d98-acb5-1869f637f4e4","Type":"ContainerDied","Data":"0a60efcd0e0dee670821e456eb39b457a02047196fc4fbb69cb5e5ec40563061"}
Jul 06 21:59:07.534585 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.534572    2582 scope.go:117] "RemoveContainer" containerID="437e273802fa35242806ee3ed9ba80b85be1c5fb5a97a6701fb89d93f48d3e1e"
Jul 06 21:59:07.543202 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.543183    2582 scope.go:117] "RemoveContainer" containerID="4b2dbd22cb1f0d714686b6e0b70325d1369ddff019f3bb79ad49a63ab053a21b"
Jul 06 21:59:07.559033 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.559007    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"]
Jul 06 21:59:07.563131 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:07.563111    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg"]
Jul 06 21:59:09.332816 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:09.332787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:09.336928 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:09.336903    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" path="/var/lib/kubelet/pods/60035eb5-7f1e-4d98-acb5-1869f637f4e4/volumes"
Jul 06 21:59:18.550254 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.550215    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"]
Jul 06 21:59:18.550748 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.550731    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerName="storage-initializer"
Jul 06 21:59:18.550803 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.550752    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerName="storage-initializer"
Jul 06 21:59:18.550803 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.550773    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerName="main"
Jul 06 21:59:18.550803 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.550781    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerName="main"
Jul 06 21:59:18.550911 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.550899    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60035eb5-7f1e-4d98-acb5-1869f637f4e4" containerName="main"
Jul 06 21:59:18.553468 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.553448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.556363 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.556344    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 06 21:59:18.564125 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.564096    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"]
Jul 06 21:59:18.584089 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.584056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:18.647161 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.647122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qgd\" (UniqueName: \"kubernetes.io/projected/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-kube-api-access-l8qgd\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.647343 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.647168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.647343 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.647259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.647343 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.647304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.647343 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.647338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.647510 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.647373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.748904 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.748869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.748943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qgd\" (UniqueName: \"kubernetes.io/projected/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-kube-api-access-l8qgd\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.748999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.749072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749115 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.749110    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749347 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.749139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749640 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.749613    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749733 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.749708    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.749789 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.749718    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.751726 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.751700    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.752277 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.752255    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.764216 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.764191    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qgd\" (UniqueName: \"kubernetes.io/projected/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-kube-api-access-l8qgd\") pod \"rolling-upgrade-test-kserve-54bc76f55b-wf42c\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:18.784313 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.784284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:18.883229 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:18.883133    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:19.033484 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:19.033454    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"]
Jul 06 21:59:19.035390 ip-10-0-131-57 kubenswrapper[2582]: W0706 21:59:19.035365    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bdf229_ca9a_4a27_8e16_3d4effeb4edc.slice/crio-afdad971d047ff2a7740b272be411e65af767b126888562e36cb8c3a96465c50 WatchSource:0}: Error finding container afdad971d047ff2a7740b272be411e65af767b126888562e36cb8c3a96465c50: Status 404 returned error can't find the container with id afdad971d047ff2a7740b272be411e65af767b126888562e36cb8c3a96465c50
Jul 06 21:59:19.604469 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:19.604436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c" event={"ID":"28bdf229-ca9a-4a27-8e16-3d4effeb4edc","Type":"ContainerStarted","Data":"a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884"}
Jul 06 21:59:19.604868 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:19.604493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c" event={"ID":"28bdf229-ca9a-4a27-8e16-3d4effeb4edc","Type":"ContainerStarted","Data":"afdad971d047ff2a7740b272be411e65af767b126888562e36cb8c3a96465c50"}
Jul 06 21:59:19.623737 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:19.623685    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c" podStartSLOduration=1.623670102 podStartE2EDuration="1.623670102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:59:19.621903558 +0000 UTC m=+5056.956244178" watchObservedRunningTime="2026-07-06 21:59:19.623670102 +0000 UTC m=+5056.958010698"
Jul 06 21:59:19.883722 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:19.883631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:27.264815 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:27.264783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:27.863939 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:27.863904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:28.883471 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:28.883433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:28.896519 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:28.896490    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:29.670769 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:29.670742    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 21:59:33.500368 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:33.500340    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:34.963098 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:34.963058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:34.965208 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:34.965180    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"]
Jul 06 21:59:34.974298 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:59:34.974263    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:59:34.974298 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:59:34.974298 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 21:59:34.974298 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686" cmd=["/bin/sleep","15"]
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:34.974323    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" message="" err=<
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:59:34.974340    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" podUID="a311dd91-c18b-40a3-8942-ab9407d33d99" containerName="main" containerID="cri-o://0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686"
Jul 06 21:59:34.974574 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:34.974412    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" podUID="a311dd91-c18b-40a3-8942-ab9407d33d99" containerName="main" containerID="cri-o://0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686" gracePeriod=60
Jul 06 21:59:35.028666 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.028637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 21:59:35.245094 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.245065    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:35.413445 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.413407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhcxv\" (UniqueName: \"kubernetes.io/projected/a311dd91-c18b-40a3-8942-ab9407d33d99-kube-api-access-zhcxv\") pod \"a311dd91-c18b-40a3-8942-ab9407d33d99\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") "
Jul 06 21:59:35.413642 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.413540    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a311dd91-c18b-40a3-8942-ab9407d33d99-tokenizer-uds\") pod \"a311dd91-c18b-40a3-8942-ab9407d33d99\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") "
Jul 06 21:59:35.413642 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.413583    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a311dd91-c18b-40a3-8942-ab9407d33d99-tls-certs\") pod \"a311dd91-c18b-40a3-8942-ab9407d33d99\" (UID: \"a311dd91-c18b-40a3-8942-ab9407d33d99\") "
Jul 06 21:59:35.413819 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.413795    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a311dd91-c18b-40a3-8942-ab9407d33d99-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a311dd91-c18b-40a3-8942-ab9407d33d99" (UID: "a311dd91-c18b-40a3-8942-ab9407d33d99"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:35.413893 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.413878    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a311dd91-c18b-40a3-8942-ab9407d33d99-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:35.415581 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.415557    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a311dd91-c18b-40a3-8942-ab9407d33d99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a311dd91-c18b-40a3-8942-ab9407d33d99" (UID: "a311dd91-c18b-40a3-8942-ab9407d33d99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:59:35.415658 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.415599    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a311dd91-c18b-40a3-8942-ab9407d33d99-kube-api-access-zhcxv" (OuterVolumeSpecName: "kube-api-access-zhcxv") pod "a311dd91-c18b-40a3-8942-ab9407d33d99" (UID: "a311dd91-c18b-40a3-8942-ab9407d33d99"). InnerVolumeSpecName "kube-api-access-zhcxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:59:35.515278 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.515173    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a311dd91-c18b-40a3-8942-ab9407d33d99-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:35.515278 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.515231    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcxv\" (UniqueName: \"kubernetes.io/projected/a311dd91-c18b-40a3-8942-ab9407d33d99-kube-api-access-zhcxv\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 21:59:35.693044 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.693008    2582 generic.go:356] "Generic (PLEG): container finished" podID="a311dd91-c18b-40a3-8942-ab9407d33d99" containerID="0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686" exitCode=0
Jul 06 21:59:35.693215 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.693059    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"
Jul 06 21:59:35.693215 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.693096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" event={"ID":"a311dd91-c18b-40a3-8942-ab9407d33d99","Type":"ContainerDied","Data":"0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686"}
Jul 06 21:59:35.693215 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.693149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t" event={"ID":"a311dd91-c18b-40a3-8942-ab9407d33d99","Type":"ContainerDied","Data":"15db0a5f0283b5f71cbf966edd46bc25cdc9ab5748d0476a38d749c3e2d1ad79"}
Jul 06 21:59:35.693215 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.693169    2582 scope.go:117] "RemoveContainer" containerID="0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686"
Jul 06 21:59:35.702927 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.702901    2582 scope.go:117] "RemoveContainer" containerID="0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686"
Jul 06 21:59:35.703260 ip-10-0-131-57 kubenswrapper[2582]: E0706 21:59:35.703236    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686\": container with ID starting with 0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686 not found: ID does not exist" containerID="0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686"
Jul 06 21:59:35.703330 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.703272    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686"} err="failed to get container status \"0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686\": rpc error: code = NotFound desc = could not find container \"0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686\": container with ID starting with 0647b9a99741ea0783c1b25dcf48b6326dfe5343bff586c5101c4d2f1b6d8686 not found: ID does not exist"
Jul 06 21:59:35.715393 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.715362    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"]
Jul 06 21:59:35.719653 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:35.719632    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t"]
Jul 06 21:59:37.337667 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:37.337634    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a311dd91-c18b-40a3-8942-ab9407d33d99" path="/var/lib/kubelet/pods/a311dd91-c18b-40a3-8942-ab9407d33d99/volumes"
Jul 06 21:59:47.684820 ip-10-0-131-57 kubenswrapper[2582]: I0706 21:59:47.684784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:02.170553 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:02.170517    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"]
Jul 06 22:00:03.814784 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:03.814747    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:00:03.834307 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:03.834286    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:00:07.269386 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:07.269339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:17.191162 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.191101    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c" podUID="28bdf229-ca9a-4a27-8e16-3d4effeb4edc" containerName="main" containerID="cri-o://a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884" gracePeriod=45
Jul 06 22:00:17.449514 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.449457    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 22:00:17.532537 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532488    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tmp-dir\") pod \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") "
Jul 06 22:00:17.532537 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8qgd\" (UniqueName: \"kubernetes.io/projected/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-kube-api-access-l8qgd\") pod \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") "
Jul 06 22:00:17.532812 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532600    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-home\") pod \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") "
Jul 06 22:00:17.532812 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532620    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tls-certs\") pod \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") "
Jul 06 22:00:17.532812 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532634    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-dshm\") pod \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") "
Jul 06 22:00:17.532812 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532690    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-model-cache\") pod \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\" (UID: \"28bdf229-ca9a-4a27-8e16-3d4effeb4edc\") "
Jul 06 22:00:17.532812 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532766    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28bdf229-ca9a-4a27-8e16-3d4effeb4edc" (UID: "28bdf229-ca9a-4a27-8e16-3d4effeb4edc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:00:17.533085 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532856    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-home" (OuterVolumeSpecName: "home") pod "28bdf229-ca9a-4a27-8e16-3d4effeb4edc" (UID: "28bdf229-ca9a-4a27-8e16-3d4effeb4edc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:00:17.533085 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.532996    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-model-cache" (OuterVolumeSpecName: "model-cache") pod "28bdf229-ca9a-4a27-8e16-3d4effeb4edc" (UID: "28bdf229-ca9a-4a27-8e16-3d4effeb4edc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:00:17.533085 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.533023    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:17.533085 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.533049    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:17.534817 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.534788    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-dshm" (OuterVolumeSpecName: "dshm") pod "28bdf229-ca9a-4a27-8e16-3d4effeb4edc" (UID: "28bdf229-ca9a-4a27-8e16-3d4effeb4edc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:00:17.534925 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.534849    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-kube-api-access-l8qgd" (OuterVolumeSpecName: "kube-api-access-l8qgd") pod "28bdf229-ca9a-4a27-8e16-3d4effeb4edc" (UID: "28bdf229-ca9a-4a27-8e16-3d4effeb4edc"). InnerVolumeSpecName "kube-api-access-l8qgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:00:17.534925 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.534857    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28bdf229-ca9a-4a27-8e16-3d4effeb4edc" (UID: "28bdf229-ca9a-4a27-8e16-3d4effeb4edc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:00:17.634103 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.634058    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:17.634103 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.634096    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:17.634103 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.634105    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:17.634103 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.634114    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qgd\" (UniqueName: \"kubernetes.io/projected/28bdf229-ca9a-4a27-8e16-3d4effeb4edc-kube-api-access-l8qgd\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:17.937211 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.937174    2582 generic.go:356] "Generic (PLEG): container finished" podID="28bdf229-ca9a-4a27-8e16-3d4effeb4edc" containerID="a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884" exitCode=0
Jul 06 22:00:17.937358 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.937224    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"
Jul 06 22:00:17.937358 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.937254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c" event={"ID":"28bdf229-ca9a-4a27-8e16-3d4effeb4edc","Type":"ContainerDied","Data":"a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884"}
Jul 06 22:00:17.937358 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.937312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c" event={"ID":"28bdf229-ca9a-4a27-8e16-3d4effeb4edc","Type":"ContainerDied","Data":"afdad971d047ff2a7740b272be411e65af767b126888562e36cb8c3a96465c50"}
Jul 06 22:00:17.937358 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.937334    2582 scope.go:117] "RemoveContainer" containerID="a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884"
Jul 06 22:00:17.947508 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.947470    2582 scope.go:117] "RemoveContainer" containerID="a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884"
Jul 06 22:00:17.947780 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:00:17.947761    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884\": container with ID starting with a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884 not found: ID does not exist" containerID="a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884"
Jul 06 22:00:17.947853 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.947789    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884"} err="failed to get container status \"a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884\": rpc error: code = NotFound desc = could not find container \"a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884\": container with ID starting with a5434a4db27ececd113291e1470e9e088887451454b949e14b9e337397169884 not found: ID does not exist"
Jul 06 22:00:17.961600 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.961575    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"]
Jul 06 22:00:17.965557 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:17.965524    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c"]
Jul 06 22:00:19.337162 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:19.337130    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28bdf229-ca9a-4a27-8e16-3d4effeb4edc" path="/var/lib/kubelet/pods/28bdf229-ca9a-4a27-8e16-3d4effeb4edc/volumes"
Jul 06 22:00:23.298034 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.297996    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"]
Jul 06 22:00:23.298417 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.298403    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a311dd91-c18b-40a3-8942-ab9407d33d99" containerName="main"
Jul 06 22:00:23.298417 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.298414    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a311dd91-c18b-40a3-8942-ab9407d33d99" containerName="main"
Jul 06 22:00:23.298493 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.298435    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28bdf229-ca9a-4a27-8e16-3d4effeb4edc" containerName="main"
Jul 06 22:00:23.298493 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.298441    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="28bdf229-ca9a-4a27-8e16-3d4effeb4edc" containerName="main"
Jul 06 22:00:23.298557 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.298508    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a311dd91-c18b-40a3-8942-ab9407d33d99" containerName="main"
Jul 06 22:00:23.298557 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.298516    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="28bdf229-ca9a-4a27-8e16-3d4effeb4edc" containerName="main"
Jul 06 22:00:23.303092 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.303072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.307382 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.307360    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 06 22:00:23.316135 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.316114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 22:00:23.319812 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.319790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"]
Jul 06 22:00:23.388338 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.388306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.388338 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.388346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.388612 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.388430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqp6m\" (UniqueName: \"kubernetes.io/projected/894ab55f-1512-4d20-bf74-46d812142d5d-kube-api-access-mqp6m\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.388612 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.388504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894ab55f-1512-4d20-bf74-46d812142d5d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.388612 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.388582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.388782 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.388696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.489581 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.489547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqp6m\" (UniqueName: \"kubernetes.io/projected/894ab55f-1512-4d20-bf74-46d812142d5d-kube-api-access-mqp6m\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.489771 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.489610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894ab55f-1512-4d20-bf74-46d812142d5d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.489771 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.489648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.489771 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.489686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.489771 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.489708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.489771 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.489736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.490071 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.490050    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.490151 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.490128    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.490191 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.490158    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.491973 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.491929    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.492373 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.492349    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894ab55f-1512-4d20-bf74-46d812142d5d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.499232 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.499207    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqp6m\" (UniqueName: \"kubernetes.io/projected/894ab55f-1512-4d20-bf74-46d812142d5d-kube-api-access-mqp6m\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.580250 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.580169    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"]
Jul 06 22:00:23.584050 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.584031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.587121 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.587100    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-9tgx2\""
Jul 06 22:00:23.594259 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.594233    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"]
Jul 06 22:00:23.613869 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.613840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:23.691095 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.690991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkshw\" (UniqueName: \"kubernetes.io/projected/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-kube-api-access-gkshw\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.691095 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.691038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.691348 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.691134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.758998 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.758954    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"]
Jul 06 22:00:23.760069 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:00:23.760044    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894ab55f_1512_4d20_bf74_46d812142d5d.slice/crio-70951370ea358733a484b85a9318d7b00eaeba9a122309ab9dbd400d12a2ec50 WatchSource:0}: Error finding container 70951370ea358733a484b85a9318d7b00eaeba9a122309ab9dbd400d12a2ec50: Status 404 returned error can't find the container with id 70951370ea358733a484b85a9318d7b00eaeba9a122309ab9dbd400d12a2ec50
Jul 06 22:00:23.792044 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.792018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkshw\" (UniqueName: \"kubernetes.io/projected/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-kube-api-access-gkshw\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.792156 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.792059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.792156 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.792148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.792513 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.792489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.794693 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.794666    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.801436 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.801413    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkshw\" (UniqueName: \"kubernetes.io/projected/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-kube-api-access-gkshw\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.895395 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.895345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:23.980132 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.980088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g" event={"ID":"894ab55f-1512-4d20-bf74-46d812142d5d","Type":"ContainerStarted","Data":"7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35"}
Jul 06 22:00:23.980295 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:23.980160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g" event={"ID":"894ab55f-1512-4d20-bf74-46d812142d5d","Type":"ContainerStarted","Data":"70951370ea358733a484b85a9318d7b00eaeba9a122309ab9dbd400d12a2ec50"}
Jul 06 22:00:24.000691 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.000537    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g" podStartSLOduration=1.000514852 podStartE2EDuration="1.000514852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:00:24.000475374 +0000 UTC m=+5121.334816004" watchObservedRunningTime="2026-07-06 22:00:24.000514852 +0000 UTC m=+5121.334855449"
Jul 06 22:00:24.042124 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.042099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"]
Jul 06 22:00:24.043474 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:00:24.043446    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6092ce8_d2f9_4a33_95ba_afe6c2e8facd.slice/crio-802841e26d6bf7c031ff5a958d13ca7168b81b4399a4ccd9cf650d38fd2f05fc WatchSource:0}: Error finding container 802841e26d6bf7c031ff5a958d13ca7168b81b4399a4ccd9cf650d38fd2f05fc: Status 404 returned error can't find the container with id 802841e26d6bf7c031ff5a958d13ca7168b81b4399a4ccd9cf650d38fd2f05fc
Jul 06 22:00:24.395927 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.395828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:24.496231 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.496195    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:24.614650 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.614564    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:24.987978 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.987936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" event={"ID":"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd","Type":"ContainerStarted","Data":"665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169"}
Jul 06 22:00:24.988213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.987996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" event={"ID":"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd","Type":"ContainerStarted","Data":"802841e26d6bf7c031ff5a958d13ca7168b81b4399a4ccd9cf650d38fd2f05fc"}
Jul 06 22:00:24.988213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:24.988056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:25.018537 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:25.018488    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" podStartSLOduration=2.018472659 podStartE2EDuration="2.018472659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:00:25.017856399 +0000 UTC m=+5122.352196998" watchObservedRunningTime="2026-07-06 22:00:25.018472659 +0000 UTC m=+5122.352813254"
Jul 06 22:00:25.999083 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:25.998941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:26.064038 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:26.064004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:27.863927 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:27.863898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:33.614367 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:33.614326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:33.627094 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:33.627070    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:34.053251 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:34.053224    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:00:55.997231 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:55.997200    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:57.256740 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.256705    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:00:57.262771 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.262736    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"]
Jul 06 22:00:57.270045 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.270018    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"]
Jul 06 22:00:57.273601 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:00:57.273566    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:00:57.273601 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:00:57.273601 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:00:57.273601 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169" cmd=["/bin/sleep","15"]
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.273622    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" message="" err=<
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:00:57.273640    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" podUID="d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" containerName="main" containerID="cri-o://665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169"
Jul 06 22:00:57.273816 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.273697    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" podUID="d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" containerName="main" containerID="cri-o://665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169" gracePeriod=60
Jul 06 22:00:57.563812 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.563788    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:57.628935 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.628903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tokenizer-uds\") pod \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") "
Jul 06 22:00:57.629145 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.628943    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tls-certs\") pod \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") "
Jul 06 22:00:57.629145 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.629046    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkshw\" (UniqueName: \"kubernetes.io/projected/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-kube-api-access-gkshw\") pod \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\" (UID: \"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd\") "
Jul 06 22:00:57.629279 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.629189    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" (UID: "d6092ce8-d2f9-4a33-95ba-afe6c2e8facd"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:00:57.629279 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.629266    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:57.631137 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.631111    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" (UID: "d6092ce8-d2f9-4a33-95ba-afe6c2e8facd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:00:57.631281 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.631222    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-kube-api-access-gkshw" (OuterVolumeSpecName: "kube-api-access-gkshw") pod "d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" (UID: "d6092ce8-d2f9-4a33-95ba-afe6c2e8facd"). InnerVolumeSpecName "kube-api-access-gkshw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:00:57.729947 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.729908    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:57.729947 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:57.729933    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkshw\" (UniqueName: \"kubernetes.io/projected/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd-kube-api-access-gkshw\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:00:58.186070 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.186037    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" containerID="665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169" exitCode=0
Jul 06 22:00:58.186255 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.186085    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"
Jul 06 22:00:58.186255 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.186133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" event={"ID":"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd","Type":"ContainerDied","Data":"665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169"}
Jul 06 22:00:58.186255 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.186180    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp" event={"ID":"d6092ce8-d2f9-4a33-95ba-afe6c2e8facd","Type":"ContainerDied","Data":"802841e26d6bf7c031ff5a958d13ca7168b81b4399a4ccd9cf650d38fd2f05fc"}
Jul 06 22:00:58.186255 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.186203    2582 scope.go:117] "RemoveContainer" containerID="665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169"
Jul 06 22:00:58.196332 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.196315    2582 scope.go:117] "RemoveContainer" containerID="665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169"
Jul 06 22:00:58.196642 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:00:58.196620    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169\": container with ID starting with 665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169 not found: ID does not exist" containerID="665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169"
Jul 06 22:00:58.196742 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.196647    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169"} err="failed to get container status \"665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169\": rpc error: code = NotFound desc = could not find container \"665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169\": container with ID starting with 665e63976f72a3d66ed2af4935c38c4636199d32e68a166f1eb4ee4447768169 not found: ID does not exist"
Jul 06 22:00:58.210834 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.210807    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"]
Jul 06 22:00:58.214713 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:58.214688    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp"]
Jul 06 22:00:59.337360 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:59.337326    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" path="/var/lib/kubelet/pods/d6092ce8-d2f9-4a33-95ba-afe6c2e8facd/volumes"
Jul 06 22:00:59.646331 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:00:59.646215    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:01:12.293164 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.293102    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g" podUID="894ab55f-1512-4d20-bf74-46d812142d5d" containerName="main" containerID="cri-o://7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35" gracePeriod=45
Jul 06 22:01:12.547564 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.547499    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:01:12.649772 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.649721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-home\") pod \"894ab55f-1512-4d20-bf74-46d812142d5d\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") "
Jul 06 22:01:12.649772 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.649788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-model-cache\") pod \"894ab55f-1512-4d20-bf74-46d812142d5d\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") "
Jul 06 22:01:12.650073 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.649819    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894ab55f-1512-4d20-bf74-46d812142d5d-tls-certs\") pod \"894ab55f-1512-4d20-bf74-46d812142d5d\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") "
Jul 06 22:01:12.650073 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.649841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-tmp-dir\") pod \"894ab55f-1512-4d20-bf74-46d812142d5d\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") "
Jul 06 22:01:12.650073 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.649888    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqp6m\" (UniqueName: \"kubernetes.io/projected/894ab55f-1512-4d20-bf74-46d812142d5d-kube-api-access-mqp6m\") pod \"894ab55f-1512-4d20-bf74-46d812142d5d\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") "
Jul 06 22:01:12.650073 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.649905    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-dshm\") pod \"894ab55f-1512-4d20-bf74-46d812142d5d\" (UID: \"894ab55f-1512-4d20-bf74-46d812142d5d\") "
Jul 06 22:01:12.650290 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.650100    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "894ab55f-1512-4d20-bf74-46d812142d5d" (UID: "894ab55f-1512-4d20-bf74-46d812142d5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:12.650290 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.650107    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-home" (OuterVolumeSpecName: "home") pod "894ab55f-1512-4d20-bf74-46d812142d5d" (UID: "894ab55f-1512-4d20-bf74-46d812142d5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:12.650290 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.650152    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "894ab55f-1512-4d20-bf74-46d812142d5d" (UID: "894ab55f-1512-4d20-bf74-46d812142d5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:12.650290 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.650201    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:12.650290 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.650221    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:12.652126 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.652100    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-dshm" (OuterVolumeSpecName: "dshm") pod "894ab55f-1512-4d20-bf74-46d812142d5d" (UID: "894ab55f-1512-4d20-bf74-46d812142d5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:12.652213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.652139    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894ab55f-1512-4d20-bf74-46d812142d5d-kube-api-access-mqp6m" (OuterVolumeSpecName: "kube-api-access-mqp6m") pod "894ab55f-1512-4d20-bf74-46d812142d5d" (UID: "894ab55f-1512-4d20-bf74-46d812142d5d"). InnerVolumeSpecName "kube-api-access-mqp6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:01:12.652254 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.652224    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894ab55f-1512-4d20-bf74-46d812142d5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "894ab55f-1512-4d20-bf74-46d812142d5d" (UID: "894ab55f-1512-4d20-bf74-46d812142d5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:01:12.751555 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.751509    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894ab55f-1512-4d20-bf74-46d812142d5d-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:12.751555 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.751550    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:12.751555 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.751563    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqp6m\" (UniqueName: \"kubernetes.io/projected/894ab55f-1512-4d20-bf74-46d812142d5d-kube-api-access-mqp6m\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:12.751798 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:12.751574    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894ab55f-1512-4d20-bf74-46d812142d5d-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:13.285741 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.285701    2582 generic.go:356] "Generic (PLEG): container finished" podID="894ab55f-1512-4d20-bf74-46d812142d5d" containerID="7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35" exitCode=0
Jul 06 22:01:13.285919 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.285753    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"
Jul 06 22:01:13.285919 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.285787    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g" event={"ID":"894ab55f-1512-4d20-bf74-46d812142d5d","Type":"ContainerDied","Data":"7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35"}
Jul 06 22:01:13.285919 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.285862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g" event={"ID":"894ab55f-1512-4d20-bf74-46d812142d5d","Type":"ContainerDied","Data":"70951370ea358733a484b85a9318d7b00eaeba9a122309ab9dbd400d12a2ec50"}
Jul 06 22:01:13.285919 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.285887    2582 scope.go:117] "RemoveContainer" containerID="7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35"
Jul 06 22:01:13.296288 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.296115    2582 scope.go:117] "RemoveContainer" containerID="7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35"
Jul 06 22:01:13.296521 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:01:13.296397    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35\": container with ID starting with 7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35 not found: ID does not exist" containerID="7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35"
Jul 06 22:01:13.296521 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.296421    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35"} err="failed to get container status \"7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35\": rpc error: code = NotFound desc = could not find container \"7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35\": container with ID starting with 7a1169e290c316705fe9131d848a3777261e651dd5bce507bb5ba8bdf8eadc35 not found: ID does not exist"
Jul 06 22:01:13.308940 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.308911    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"]
Jul 06 22:01:13.312993 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.312939    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g"]
Jul 06 22:01:13.337823 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:13.337794    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894ab55f-1512-4d20-bf74-46d812142d5d" path="/var/lib/kubelet/pods/894ab55f-1512-4d20-bf74-46d812142d5d/volumes"
Jul 06 22:01:19.398025 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.397991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:01:19.676898 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.676857    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"]
Jul 06 22:01:19.677888 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.677861    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894ab55f-1512-4d20-bf74-46d812142d5d" containerName="main"
Jul 06 22:01:19.678013 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.677890    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="894ab55f-1512-4d20-bf74-46d812142d5d" containerName="main"
Jul 06 22:01:19.678013 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.677929    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" containerName="main"
Jul 06 22:01:19.678013 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.677938    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" containerName="main"
Jul 06 22:01:19.678177 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.678160    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="894ab55f-1512-4d20-bf74-46d812142d5d" containerName="main"
Jul 06 22:01:19.678225 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.678191    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6092ce8-d2f9-4a33-95ba-afe6c2e8facd" containerName="main"
Jul 06 22:01:19.687559 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.687529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.691275 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.691245    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"]
Jul 06 22:01:19.694206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.694025    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 22:01:19.694206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.694067    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 06 22:01:19.694206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.694125    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-dwz2w\""
Jul 06 22:01:19.715537 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.715507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gz2\" (UniqueName: \"kubernetes.io/projected/eeed94c7-09d9-4629-b235-1e82cba40d65-kube-api-access-w9gz2\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.715689 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.715557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eeed94c7-09d9-4629-b235-1e82cba40d65-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.715689 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.715592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed94c7-09d9-4629-b235-1e82cba40d65-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.734299 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.734274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:01:19.816356 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.816323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gz2\" (UniqueName: \"kubernetes.io/projected/eeed94c7-09d9-4629-b235-1e82cba40d65-kube-api-access-w9gz2\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.816538 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.816374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eeed94c7-09d9-4629-b235-1e82cba40d65-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.816538 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.816400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed94c7-09d9-4629-b235-1e82cba40d65-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.816743 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.816724    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eeed94c7-09d9-4629-b235-1e82cba40d65-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.818838 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.818809    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed94c7-09d9-4629-b235-1e82cba40d65-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:19.835584 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:19.835554    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gz2\" (UniqueName: \"kubernetes.io/projected/eeed94c7-09d9-4629-b235-1e82cba40d65-kube-api-access-w9gz2\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:20.000474 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:20.000385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:20.147950 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:20.147919    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"]
Jul 06 22:01:20.149376 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:01:20.149346    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeed94c7_09d9_4629_b235_1e82cba40d65.slice/crio-0601bab86b1dbc4dcb910b6ef17c847ddd325a203cc89781b6685c8c4c910b45 WatchSource:0}: Error finding container 0601bab86b1dbc4dcb910b6ef17c847ddd325a203cc89781b6685c8c4c910b45: Status 404 returned error can't find the container with id 0601bab86b1dbc4dcb910b6ef17c847ddd325a203cc89781b6685c8c4c910b45
Jul 06 22:01:20.327724 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:20.327625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" event={"ID":"eeed94c7-09d9-4629-b235-1e82cba40d65","Type":"ContainerStarted","Data":"c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd"}
Jul 06 22:01:20.327724 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:20.327695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" event={"ID":"eeed94c7-09d9-4629-b235-1e82cba40d65","Type":"ContainerStarted","Data":"0601bab86b1dbc4dcb910b6ef17c847ddd325a203cc89781b6685c8c4c910b45"}
Jul 06 22:01:20.327950 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:20.327880    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:01:20.352934 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:20.352868    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" podStartSLOduration=1.352846659 podStartE2EDuration="1.352846659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:01:20.352396571 +0000 UTC m=+5177.686737168" watchObservedRunningTime="2026-07-06 22:01:20.352846659 +0000 UTC m=+5177.687187255"
Jul 06 22:01:22.993237 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:22.993202    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7f65c45d5c-7t766"]
Jul 06 22:01:22.993649 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:22.993461    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766" podUID="2040c113-44c2-40c4-acf0-0cc9749c489a" containerName="manager" containerID="cri-o://d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49" gracePeriod=30
Jul 06 22:01:39.148634 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.148607    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 22:01:39.296111 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.296023    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m5bl\" (UniqueName: \"kubernetes.io/projected/2040c113-44c2-40c4-acf0-0cc9749c489a-kube-api-access-7m5bl\") pod \"2040c113-44c2-40c4-acf0-0cc9749c489a\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") "
Jul 06 22:01:39.296111 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.296093    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2040c113-44c2-40c4-acf0-0cc9749c489a-cert\") pod \"2040c113-44c2-40c4-acf0-0cc9749c489a\" (UID: \"2040c113-44c2-40c4-acf0-0cc9749c489a\") "
Jul 06 22:01:39.298233 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.298211    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2040c113-44c2-40c4-acf0-0cc9749c489a-cert" (OuterVolumeSpecName: "cert") pod "2040c113-44c2-40c4-acf0-0cc9749c489a" (UID: "2040c113-44c2-40c4-acf0-0cc9749c489a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:01:39.298318 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.298304    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2040c113-44c2-40c4-acf0-0cc9749c489a-kube-api-access-7m5bl" (OuterVolumeSpecName: "kube-api-access-7m5bl") pod "2040c113-44c2-40c4-acf0-0cc9749c489a" (UID: "2040c113-44c2-40c4-acf0-0cc9749c489a"). InnerVolumeSpecName "kube-api-access-7m5bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:01:39.397941 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.397902    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m5bl\" (UniqueName: \"kubernetes.io/projected/2040c113-44c2-40c4-acf0-0cc9749c489a-kube-api-access-7m5bl\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:39.397941 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.397936    2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2040c113-44c2-40c4-acf0-0cc9749c489a-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:01:39.432327 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.432287    2582 generic.go:356] "Generic (PLEG): container finished" podID="2040c113-44c2-40c4-acf0-0cc9749c489a" containerID="d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49" exitCode=0
Jul 06 22:01:39.432490 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.432352    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766"
Jul 06 22:01:39.432490 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.432376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766" event={"ID":"2040c113-44c2-40c4-acf0-0cc9749c489a","Type":"ContainerDied","Data":"d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49"}
Jul 06 22:01:39.432490 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.432431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7f65c45d5c-7t766" event={"ID":"2040c113-44c2-40c4-acf0-0cc9749c489a","Type":"ContainerDied","Data":"19b3905fdb40f8b982636f3201bbd8ca4d03667b28d23a8c48f539c0ce723497"}
Jul 06 22:01:39.432490 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.432457    2582 scope.go:117] "RemoveContainer" containerID="d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49"
Jul 06 22:01:39.443227 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.443207    2582 scope.go:117] "RemoveContainer" containerID="d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49"
Jul 06 22:01:39.443489 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:01:39.443470    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49\": container with ID starting with d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49 not found: ID does not exist" containerID="d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49"
Jul 06 22:01:39.443540 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.443499    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49"} err="failed to get container status \"d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49\": rpc error: code = NotFound desc = could not find container \"d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49\": container with ID starting with d2c990a78fb5d91298ac1b729fa43296901fcfa555cc8fa7b9d6d815a252ac49 not found: ID does not exist"
Jul 06 22:01:39.454613 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.454586    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7f65c45d5c-7t766"]
Jul 06 22:01:39.457554 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:39.457531    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-7f65c45d5c-7t766"]
Jul 06 22:01:41.337373 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:41.337339    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2040c113-44c2-40c4-acf0-0cc9749c489a" path="/var/lib/kubelet/pods/2040c113-44c2-40c4-acf0-0cc9749c489a/volumes"
Jul 06 22:01:47.068684 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:47.068652    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-lw446"]
Jul 06 22:01:47.073922 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:47.073897    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-lw446"]
Jul 06 22:01:47.337104 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:47.337006    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd553b25-654c-43bd-a8d5-14ebc59a4dc8" path="/var/lib/kubelet/pods/dd553b25-654c-43bd-a8d5-14ebc59a4dc8/volumes"
Jul 06 22:01:51.338693 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:01:51.338661    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:02:04.113110 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:04.113078    2582 scope.go:117] "RemoveContainer" containerID="0b7729ca71af2c329f90b283b7f3bb5c6304bd89c1c728b453fb2148fa81b488"
Jul 06 22:02:08.084443 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:08.084410    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:10.111977 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.111927    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"]
Jul 06 22:02:10.118034 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.118004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:10.122343 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:02:10.122308    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:02:10.122343 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:02:10.122343 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:02:10.122343 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd" cmd=["/bin/sleep","15"]
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.122367    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" message="" err=<
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:02:10.122384    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" podUID="eeed94c7-09d9-4629-b235-1e82cba40d65" containerName="main" containerID="cri-o://c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd"
Jul 06 22:02:10.122612 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.122449    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" podUID="eeed94c7-09d9-4629-b235-1e82cba40d65" containerName="main" containerID="cri-o://c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd" gracePeriod=60
Jul 06 22:02:10.213982 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.213934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:10.376850 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.376775    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:02:10.497426 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.497394    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9gz2\" (UniqueName: \"kubernetes.io/projected/eeed94c7-09d9-4629-b235-1e82cba40d65-kube-api-access-w9gz2\") pod \"eeed94c7-09d9-4629-b235-1e82cba40d65\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") "
Jul 06 22:02:10.497609 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.497444    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eeed94c7-09d9-4629-b235-1e82cba40d65-tokenizer-uds\") pod \"eeed94c7-09d9-4629-b235-1e82cba40d65\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") "
Jul 06 22:02:10.497609 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.497542    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed94c7-09d9-4629-b235-1e82cba40d65-tls-certs\") pod \"eeed94c7-09d9-4629-b235-1e82cba40d65\" (UID: \"eeed94c7-09d9-4629-b235-1e82cba40d65\") "
Jul 06 22:02:10.497733 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.497710    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed94c7-09d9-4629-b235-1e82cba40d65-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "eeed94c7-09d9-4629-b235-1e82cba40d65" (UID: "eeed94c7-09d9-4629-b235-1e82cba40d65"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:02:10.499683 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.499646    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeed94c7-09d9-4629-b235-1e82cba40d65-kube-api-access-w9gz2" (OuterVolumeSpecName: "kube-api-access-w9gz2") pod "eeed94c7-09d9-4629-b235-1e82cba40d65" (UID: "eeed94c7-09d9-4629-b235-1e82cba40d65"). InnerVolumeSpecName "kube-api-access-w9gz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:02:10.499801 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.499658    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeed94c7-09d9-4629-b235-1e82cba40d65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeed94c7-09d9-4629-b235-1e82cba40d65" (UID: "eeed94c7-09d9-4629-b235-1e82cba40d65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:02:10.598764 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.598731    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeed94c7-09d9-4629-b235-1e82cba40d65-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:02:10.598764 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.598761    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gz2\" (UniqueName: \"kubernetes.io/projected/eeed94c7-09d9-4629-b235-1e82cba40d65-kube-api-access-w9gz2\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:02:10.598764 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.598770    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eeed94c7-09d9-4629-b235-1e82cba40d65-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:02:10.610042 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.610011    2582 generic.go:356] "Generic (PLEG): container finished" podID="eeed94c7-09d9-4629-b235-1e82cba40d65" containerID="c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd" exitCode=0
Jul 06 22:02:10.610184 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.610061    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"
Jul 06 22:02:10.610184 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.610099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" event={"ID":"eeed94c7-09d9-4629-b235-1e82cba40d65","Type":"ContainerDied","Data":"c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd"}
Jul 06 22:02:10.610184 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.610150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb" event={"ID":"eeed94c7-09d9-4629-b235-1e82cba40d65","Type":"ContainerDied","Data":"0601bab86b1dbc4dcb910b6ef17c847ddd325a203cc89781b6685c8c4c910b45"}
Jul 06 22:02:10.610184 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.610171    2582 scope.go:117] "RemoveContainer" containerID="c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd"
Jul 06 22:02:10.620376 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.620352    2582 scope.go:117] "RemoveContainer" containerID="c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd"
Jul 06 22:02:10.620722 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:02:10.620655    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd\": container with ID starting with c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd not found: ID does not exist" containerID="c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd"
Jul 06 22:02:10.620722 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.620696    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd"} err="failed to get container status \"c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd\": rpc error: code = NotFound desc = could not find container \"c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd\": container with ID starting with c885dd784f6fe996f2a285d3d7ad6ed59d6cd0297111197e607632393b381bfd not found: ID does not exist"
Jul 06 22:02:10.634666 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.634606    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"]
Jul 06 22:02:10.638482 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:10.638455    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb"]
Jul 06 22:02:11.336522 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:11.336489    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeed94c7-09d9-4629-b235-1e82cba40d65" path="/var/lib/kubelet/pods/eeed94c7-09d9-4629-b235-1e82cba40d65/volumes"
Jul 06 22:02:14.344861 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:14.344830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:35.791719 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.791685    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"]
Jul 06 22:02:35.792206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.792093    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2040c113-44c2-40c4-acf0-0cc9749c489a" containerName="manager"
Jul 06 22:02:35.792206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.792105    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2040c113-44c2-40c4-acf0-0cc9749c489a" containerName="manager"
Jul 06 22:02:35.792206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.792127    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeed94c7-09d9-4629-b235-1e82cba40d65" containerName="main"
Jul 06 22:02:35.792206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.792132    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeed94c7-09d9-4629-b235-1e82cba40d65" containerName="main"
Jul 06 22:02:35.792206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.792198    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2040c113-44c2-40c4-acf0-0cc9749c489a" containerName="manager"
Jul 06 22:02:35.792206 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.792208    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeed94c7-09d9-4629-b235-1e82cba40d65" containerName="main"
Jul 06 22:02:35.795157 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.795140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.799903 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.799879    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 06 22:02:35.800070 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.799942    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 22:02:35.818649 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.818617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"]
Jul 06 22:02:35.833374 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.833343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6kd\" (UniqueName: \"kubernetes.io/projected/1baa8b20-6216-4588-8d52-a74e71737508-kube-api-access-4c6kd\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.833513 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.833396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.833513 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.833457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.833513 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.833485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1baa8b20-6216-4588-8d52-a74e71737508-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.833687 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.833620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.833687 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.833667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.846110 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.846083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:35.934674 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.934640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.934674 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.934680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1baa8b20-6216-4588-8d52-a74e71737508-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.934930 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.934728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.934930 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.934852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.935079 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.934936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6kd\" (UniqueName: \"kubernetes.io/projected/1baa8b20-6216-4588-8d52-a74e71737508-kube-api-access-4c6kd\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.935079 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.935003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.935262 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.935138    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.935306 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.935274    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.935359 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.935325    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.937510 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.937484    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.937886 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.937865    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1baa8b20-6216-4588-8d52-a74e71737508-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:35.949173 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:35.949144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6kd\" (UniqueName: \"kubernetes.io/projected/1baa8b20-6216-4588-8d52-a74e71737508-kube-api-access-4c6kd\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-vz68r\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:36.105533 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.105426    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:36.265368 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.262551    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"]
Jul 06 22:02:36.272505 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.272478    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"]
Jul 06 22:02:36.276054 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.276038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.280269 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.280252    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-f2vvr\""
Jul 06 22:02:36.293903 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.293879    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"]
Jul 06 22:02:36.308605 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.308574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:36.339327 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.339300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.339474 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.339356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.339474 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.339440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.339474 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.339472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.339654 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.339553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4b8x\" (UniqueName: \"kubernetes.io/projected/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-kube-api-access-l4b8x\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.440657 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.440619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.440824 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.440692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.440824 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.440736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.440824 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.440765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.441064 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.440838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4b8x\" (UniqueName: \"kubernetes.io/projected/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-kube-api-access-l4b8x\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.441134 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.441108    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.441192 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.441130    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.441192 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.441167    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.443256 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.443235    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.463995 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.463956    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4b8x\" (UniqueName: \"kubernetes.io/projected/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-kube-api-access-l4b8x\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567wx8kj\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.587991 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.587935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:36.741580 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.741557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"]
Jul 06 22:02:36.743538 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:02:36.743506    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d024d9a_f8d4_46bd_bdd3_d27d725d8280.slice/crio-8077c7b24a53109e0b9eec49ee55b09735bbfdcc17e0ce01343724da44d39b89 WatchSource:0}: Error finding container 8077c7b24a53109e0b9eec49ee55b09735bbfdcc17e0ce01343724da44d39b89: Status 404 returned error can't find the container with id 8077c7b24a53109e0b9eec49ee55b09735bbfdcc17e0ce01343724da44d39b89
Jul 06 22:02:36.757554 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.757524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" event={"ID":"3d024d9a-f8d4-46bd-bdd3-d27d725d8280","Type":"ContainerStarted","Data":"8077c7b24a53109e0b9eec49ee55b09735bbfdcc17e0ce01343724da44d39b89"}
Jul 06 22:02:36.759175 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.759149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r" event={"ID":"1baa8b20-6216-4588-8d52-a74e71737508","Type":"ContainerStarted","Data":"1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc"}
Jul 06 22:02:36.759270 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.759198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r" event={"ID":"1baa8b20-6216-4588-8d52-a74e71737508","Type":"ContainerStarted","Data":"6e9ff1718552f3054602793d6f838ffa86e240076520ddd50587f7856fc3d7b4"}
Jul 06 22:02:36.782678 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:36.782629    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r" podStartSLOduration=1.7826081010000001 podStartE2EDuration="1.782608101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:02:36.78101034 +0000 UTC m=+5254.115350940" watchObservedRunningTime="2026-07-06 22:02:36.782608101 +0000 UTC m=+5254.116948708"
Jul 06 22:02:37.106542 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:37.106451    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:37.765933 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:37.765881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" event={"ID":"3d024d9a-f8d4-46bd-bdd3-d27d725d8280","Type":"ContainerStarted","Data":"cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5"}
Jul 06 22:02:37.766138 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:37.766077    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:02:37.792058 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:37.792003    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" podStartSLOduration=1.791988546 podStartE2EDuration="1.791988546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:02:37.790421112 +0000 UTC m=+5255.124761728" watchObservedRunningTime="2026-07-06 22:02:37.791988546 +0000 UTC m=+5255.126329140"
Jul 06 22:02:46.106584 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:46.106550    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:46.119460 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:46.119436    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:46.834928 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:46.834899    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:02:50.086213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.086179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:50.260640 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.260605    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"]
Jul 06 22:02:50.264501 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.264483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.268905 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.268884    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-bkrv6\""
Jul 06 22:02:50.269056 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.268934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 06 22:02:50.284576 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.284542    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"]
Jul 06 22:02:50.302407 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.302367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:02:50.371897 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.371803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.371897 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.371853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56j8z\" (UniqueName: \"kubernetes.io/projected/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-kube-api-access-56j8z\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.372137 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.372046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.473103 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.473066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.473291 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.473124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.473291 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.473162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56j8z\" (UniqueName: \"kubernetes.io/projected/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-kube-api-access-56j8z\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.473495 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.473472    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.475708 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.475683    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.482985 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.482944    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56j8z\" (UniqueName: \"kubernetes.io/projected/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-kube-api-access-56j8z\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.578972 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.578939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.723652 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.723616    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"]
Jul 06 22:02:50.725920 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:02:50.725888    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6553ee3a_0a48_4de4_8586_ddb8a05e13a0.slice/crio-38fe633d28440cc70ba5cc4de2e0180d83274a6dd96f02441654e3fd2fded7c8 WatchSource:0}: Error finding container 38fe633d28440cc70ba5cc4de2e0180d83274a6dd96f02441654e3fd2fded7c8: Status 404 returned error can't find the container with id 38fe633d28440cc70ba5cc4de2e0180d83274a6dd96f02441654e3fd2fded7c8
Jul 06 22:02:50.848006 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.847948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" event={"ID":"6553ee3a-0a48-4de4-8586-ddb8a05e13a0","Type":"ContainerStarted","Data":"bf157f3dd70751e12c9c7c648f55f7cd35e39a51e6be83c6550598567b67e56f"}
Jul 06 22:02:50.848198 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.848025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" event={"ID":"6553ee3a-0a48-4de4-8586-ddb8a05e13a0","Type":"ContainerStarted","Data":"38fe633d28440cc70ba5cc4de2e0180d83274a6dd96f02441654e3fd2fded7c8"}
Jul 06 22:02:50.848198 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.848105    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:02:50.876873 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:02:50.876807    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" podStartSLOduration=0.87678552 podStartE2EDuration="876.78552ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:02:50.872560214 +0000 UTC m=+5268.206900809" watchObservedRunningTime="2026-07-06 22:02:50.87678552 +0000 UTC m=+5268.211126116"
Jul 06 22:03:06.601610 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:06.601572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:08.774292 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:08.774265    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:03:08.806395 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:08.806360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:08.907413 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:08.907363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:19.818429 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:19.818398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:21.854920 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:21.854891    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:03:22.352917 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.352885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:22.362312 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.362276    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"]
Jul 06 22:03:22.366159 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.366133    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"]
Jul 06 22:03:22.371562 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:03:22.371524    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:03:22.371562 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:22.371562 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:22.371562 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5" cmd=["/bin/sleep","15"]
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.371577    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" message="" err=<
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:03:22.371589    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" podUID="3d024d9a-f8d4-46bd-bdd3-d27d725d8280" containerName="main" containerID="cri-o://cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5"
Jul 06 22:03:22.371803 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.371634    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" podUID="3d024d9a-f8d4-46bd-bdd3-d27d725d8280" containerName="main" containerID="cri-o://cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5" gracePeriod=60
Jul 06 22:03:22.459660 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.459624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:22.635384 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.635355    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:03:22.675191 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-cache\") pod \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") "
Jul 06 22:03:22.675381 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-cachi2-cache\") pod \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") "
Jul 06 22:03:22.675381 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675248    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-uds\") pod \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") "
Jul 06 22:03:22.675381 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tls-certs\") pod \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") "
Jul 06 22:03:22.675570 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675463    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4b8x\" (UniqueName: \"kubernetes.io/projected/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-kube-api-access-l4b8x\") pod \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\" (UID: \"3d024d9a-f8d4-46bd-bdd3-d27d725d8280\") "
Jul 06 22:03:22.675631 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675462    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3d024d9a-f8d4-46bd-bdd3-d27d725d8280" (UID: "3d024d9a-f8d4-46bd-bdd3-d27d725d8280"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:22.675631 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675524    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "3d024d9a-f8d4-46bd-bdd3-d27d725d8280" (UID: "3d024d9a-f8d4-46bd-bdd3-d27d725d8280"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:22.675737 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675644    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3d024d9a-f8d4-46bd-bdd3-d27d725d8280" (UID: "3d024d9a-f8d4-46bd-bdd3-d27d725d8280"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:22.675848 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675828    2582 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-cachi2-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:22.675904 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675854    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:22.675904 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.675867    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tokenizer-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:22.677524 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.677493    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d024d9a-f8d4-46bd-bdd3-d27d725d8280" (UID: "3d024d9a-f8d4-46bd-bdd3-d27d725d8280"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:03:22.677816 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.677790    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-kube-api-access-l4b8x" (OuterVolumeSpecName: "kube-api-access-l4b8x") pod "3d024d9a-f8d4-46bd-bdd3-d27d725d8280" (UID: "3d024d9a-f8d4-46bd-bdd3-d27d725d8280"). InnerVolumeSpecName "kube-api-access-l4b8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:03:22.777150 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.777111    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4b8x\" (UniqueName: \"kubernetes.io/projected/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-kube-api-access-l4b8x\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:22.777150 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:22.777144    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d024d9a-f8d4-46bd-bdd3-d27d725d8280-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:23.033765 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.033731    2582 generic.go:356] "Generic (PLEG): container finished" podID="3d024d9a-f8d4-46bd-bdd3-d27d725d8280" containerID="cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5" exitCode=0
Jul 06 22:03:23.034174 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.033770    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"
Jul 06 22:03:23.034174 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.033809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" event={"ID":"3d024d9a-f8d4-46bd-bdd3-d27d725d8280","Type":"ContainerDied","Data":"cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5"}
Jul 06 22:03:23.034174 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.033859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj" event={"ID":"3d024d9a-f8d4-46bd-bdd3-d27d725d8280","Type":"ContainerDied","Data":"8077c7b24a53109e0b9eec49ee55b09735bbfdcc17e0ce01343724da44d39b89"}
Jul 06 22:03:23.034174 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.033880    2582 scope.go:117] "RemoveContainer" containerID="cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5"
Jul 06 22:03:23.045345 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.045325    2582 scope.go:117] "RemoveContainer" containerID="cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5"
Jul 06 22:03:23.045614 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:03:23.045595    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5\": container with ID starting with cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5 not found: ID does not exist" containerID="cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5"
Jul 06 22:03:23.045667 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.045623    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5"} err="failed to get container status \"cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5\": rpc error: code = NotFound desc = could not find container \"cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5\": container with ID starting with cf56d3450f865ac43f0b0fa050f8ff28ebfa4040f4c1079eead7e4ecd5e145c5 not found: ID does not exist"
Jul 06 22:03:23.059675 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.059648    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"]
Jul 06 22:03:23.065035 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.064999    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj"]
Jul 06 22:03:23.337618 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.337538    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d024d9a-f8d4-46bd-bdd3-d27d725d8280" path="/var/lib/kubelet/pods/3d024d9a-f8d4-46bd-bdd3-d27d725d8280/volumes"
Jul 06 22:03:23.776285 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.776255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:23.784548 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.784519    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"]
Jul 06 22:03:23.794241 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:03:23.794201    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:03:23.794241 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:23.794241 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:23.794241 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="bf157f3dd70751e12c9c7c648f55f7cd35e39a51e6be83c6550598567b67e56f" cmd=["/bin/sleep","15"]
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.794264    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" message="" err=<
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:03:23.794283    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" podUID="6553ee3a-0a48-4de4-8586-ddb8a05e13a0" containerName="main" containerID="cri-o://bf157f3dd70751e12c9c7c648f55f7cd35e39a51e6be83c6550598567b67e56f"
Jul 06 22:03:23.794529 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.794327    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" podUID="6553ee3a-0a48-4de4-8586-ddb8a05e13a0" containerName="main" containerID="cri-o://bf157f3dd70751e12c9c7c648f55f7cd35e39a51e6be83c6550598567b67e56f" gracePeriod=60
Jul 06 22:03:23.868921 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:23.868876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:24.041944 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.041825    2582 generic.go:356] "Generic (PLEG): container finished" podID="6553ee3a-0a48-4de4-8586-ddb8a05e13a0" containerID="bf157f3dd70751e12c9c7c648f55f7cd35e39a51e6be83c6550598567b67e56f" exitCode=0
Jul 06 22:03:24.041944 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.041898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" event={"ID":"6553ee3a-0a48-4de4-8586-ddb8a05e13a0","Type":"ContainerDied","Data":"bf157f3dd70751e12c9c7c648f55f7cd35e39a51e6be83c6550598567b67e56f"}
Jul 06 22:03:24.057813 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.057789    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:03:24.090149 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.090123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tls-certs\") pod \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") "
Jul 06 22:03:24.090318 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.090197    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tokenizer-uds\") pod \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") "
Jul 06 22:03:24.090318 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.090245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56j8z\" (UniqueName: \"kubernetes.io/projected/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-kube-api-access-56j8z\") pod \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\" (UID: \"6553ee3a-0a48-4de4-8586-ddb8a05e13a0\") "
Jul 06 22:03:24.090468 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.090448    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6553ee3a-0a48-4de4-8586-ddb8a05e13a0" (UID: "6553ee3a-0a48-4de4-8586-ddb8a05e13a0"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:24.090626 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.090610    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:24.092351 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.092329    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-kube-api-access-56j8z" (OuterVolumeSpecName: "kube-api-access-56j8z") pod "6553ee3a-0a48-4de4-8586-ddb8a05e13a0" (UID: "6553ee3a-0a48-4de4-8586-ddb8a05e13a0"). InnerVolumeSpecName "kube-api-access-56j8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:03:24.092425 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.092345    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6553ee3a-0a48-4de4-8586-ddb8a05e13a0" (UID: "6553ee3a-0a48-4de4-8586-ddb8a05e13a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:03:24.192033 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.191999    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:24.192033 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:24.192028    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56j8z\" (UniqueName: \"kubernetes.io/projected/6553ee3a-0a48-4de4-8586-ddb8a05e13a0-kube-api-access-56j8z\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:25.049534 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:25.049450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t" event={"ID":"6553ee3a-0a48-4de4-8586-ddb8a05e13a0","Type":"ContainerDied","Data":"38fe633d28440cc70ba5cc4de2e0180d83274a6dd96f02441654e3fd2fded7c8"}
Jul 06 22:03:25.049534 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:25.049508    2582 scope.go:117] "RemoveContainer" containerID="bf157f3dd70751e12c9c7c648f55f7cd35e39a51e6be83c6550598567b67e56f"
Jul 06 22:03:25.050054 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:25.049466    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"
Jul 06 22:03:25.084654 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:25.084625    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"]
Jul 06 22:03:25.091738 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:25.091711    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t"]
Jul 06 22:03:25.337915 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:25.337835    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6553ee3a-0a48-4de4-8586-ddb8a05e13a0" path="/var/lib/kubelet/pods/6553ee3a-0a48-4de4-8586-ddb8a05e13a0/volumes"
Jul 06 22:03:37.387933 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.387893    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r" podUID="1baa8b20-6216-4588-8d52-a74e71737508" containerName="main" containerID="cri-o://1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc" gracePeriod=45
Jul 06 22:03:37.644351 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.644294    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:03:37.717600 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.717571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-model-cache\") pod \"1baa8b20-6216-4588-8d52-a74e71737508\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") "
Jul 06 22:03:37.717806 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.717609    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1baa8b20-6216-4588-8d52-a74e71737508-tls-certs\") pod \"1baa8b20-6216-4588-8d52-a74e71737508\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") "
Jul 06 22:03:37.717806 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.717669    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c6kd\" (UniqueName: \"kubernetes.io/projected/1baa8b20-6216-4588-8d52-a74e71737508-kube-api-access-4c6kd\") pod \"1baa8b20-6216-4588-8d52-a74e71737508\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") "
Jul 06 22:03:37.717806 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.717695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-dshm\") pod \"1baa8b20-6216-4588-8d52-a74e71737508\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") "
Jul 06 22:03:37.717806 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.717719    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-home\") pod \"1baa8b20-6216-4588-8d52-a74e71737508\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") "
Jul 06 22:03:37.718076 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.717851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-tmp-dir\") pod \"1baa8b20-6216-4588-8d52-a74e71737508\" (UID: \"1baa8b20-6216-4588-8d52-a74e71737508\") "
Jul 06 22:03:37.718076 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.717891    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-model-cache" (OuterVolumeSpecName: "model-cache") pod "1baa8b20-6216-4588-8d52-a74e71737508" (UID: "1baa8b20-6216-4588-8d52-a74e71737508"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:37.718076 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.718020    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-home" (OuterVolumeSpecName: "home") pod "1baa8b20-6216-4588-8d52-a74e71737508" (UID: "1baa8b20-6216-4588-8d52-a74e71737508"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:37.718202 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.718100    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1baa8b20-6216-4588-8d52-a74e71737508" (UID: "1baa8b20-6216-4588-8d52-a74e71737508"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:37.718287 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.718267    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-home\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:37.718333 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.718295    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-tmp-dir\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:37.718333 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.718308    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-model-cache\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:37.719851 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.719804    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-dshm" (OuterVolumeSpecName: "dshm") pod "1baa8b20-6216-4588-8d52-a74e71737508" (UID: "1baa8b20-6216-4588-8d52-a74e71737508"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:37.719958 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.719857    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1baa8b20-6216-4588-8d52-a74e71737508-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1baa8b20-6216-4588-8d52-a74e71737508" (UID: "1baa8b20-6216-4588-8d52-a74e71737508"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:03:37.719958 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.719935    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baa8b20-6216-4588-8d52-a74e71737508-kube-api-access-4c6kd" (OuterVolumeSpecName: "kube-api-access-4c6kd") pod "1baa8b20-6216-4588-8d52-a74e71737508" (UID: "1baa8b20-6216-4588-8d52-a74e71737508"). InnerVolumeSpecName "kube-api-access-4c6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:03:37.819182 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.819143    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1baa8b20-6216-4588-8d52-a74e71737508-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:37.819182 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.819173    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6kd\" (UniqueName: \"kubernetes.io/projected/1baa8b20-6216-4588-8d52-a74e71737508-kube-api-access-4c6kd\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:37.819182 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:37.819182    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1baa8b20-6216-4588-8d52-a74e71737508-dshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:03:38.121340 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.121301    2582 generic.go:356] "Generic (PLEG): container finished" podID="1baa8b20-6216-4588-8d52-a74e71737508" containerID="1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc" exitCode=0
Jul 06 22:03:38.121551 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.121346    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"
Jul 06 22:03:38.121551 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.121350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r" event={"ID":"1baa8b20-6216-4588-8d52-a74e71737508","Type":"ContainerDied","Data":"1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc"}
Jul 06 22:03:38.121551 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.121423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r" event={"ID":"1baa8b20-6216-4588-8d52-a74e71737508","Type":"ContainerDied","Data":"6e9ff1718552f3054602793d6f838ffa86e240076520ddd50587f7856fc3d7b4"}
Jul 06 22:03:38.121551 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.121455    2582 scope.go:117] "RemoveContainer" containerID="1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc"
Jul 06 22:03:38.133642 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.133621    2582 scope.go:117] "RemoveContainer" containerID="1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc"
Jul 06 22:03:38.133915 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:03:38.133891    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc\": container with ID starting with 1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc not found: ID does not exist" containerID="1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc"
Jul 06 22:03:38.133995 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.133928    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc"} err="failed to get container status \"1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc\": rpc error: code = NotFound desc = could not find container \"1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc\": container with ID starting with 1f85c6914d3852105fe8c9e1aa2466f2ec649c6d0c78e2e0a4ccece996d1bbcc not found: ID does not exist"
Jul 06 22:03:38.145549 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.145525    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"]
Jul 06 22:03:38.147805 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:38.147788    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r"]
Jul 06 22:03:39.337145 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:39.337113    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1baa8b20-6216-4588-8d52-a74e71737508" path="/var/lib/kubelet/pods/1baa8b20-6216-4588-8d52-a74e71737508/volumes"
Jul 06 22:03:49.012873 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.012834    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"]
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013415    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1baa8b20-6216-4588-8d52-a74e71737508" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013431    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1baa8b20-6216-4588-8d52-a74e71737508" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013441    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6553ee3a-0a48-4de4-8586-ddb8a05e13a0" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013446    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6553ee3a-0a48-4de4-8586-ddb8a05e13a0" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d024d9a-f8d4-46bd-bdd3-d27d725d8280" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013469    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d024d9a-f8d4-46bd-bdd3-d27d725d8280" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013533    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d024d9a-f8d4-46bd-bdd3-d27d725d8280" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013541    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6553ee3a-0a48-4de4-8586-ddb8a05e13a0" containerName="main"
Jul 06 22:03:49.015213 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.013551    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1baa8b20-6216-4588-8d52-a74e71737508" containerName="main"
Jul 06 22:03:49.016649 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.016632    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.021009 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.020989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 06 22:03:49.021276 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.021260    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qcsh\""
Jul 06 22:03:49.021375 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.021278    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-xsff2\""
Jul 06 22:03:49.034578 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.034553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"]
Jul 06 22:03:49.073914 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.073883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:49.128477 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.128436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htfgq\" (UniqueName: \"kubernetes.io/projected/b5607a6f-b210-4689-adce-f822443fecc2-kube-api-access-htfgq\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.128651 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.128489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5607a6f-b210-4689-adce-f822443fecc2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.128651 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.128564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.128730 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.128646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.128730 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.128677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.128730 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.128712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.229827 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.229789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230032 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.229832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230032 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.229878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230032 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.229911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htfgq\" (UniqueName: \"kubernetes.io/projected/b5607a6f-b210-4689-adce-f822443fecc2-kube-api-access-htfgq\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230032 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.229949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5607a6f-b210-4689-adce-f822443fecc2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230284 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.230051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230354 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.230324    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230427 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.230337    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.230486 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.230434    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.232166 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.232148    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5607a6f-b210-4689-adce-f822443fecc2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.232372 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.232355    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5607a6f-b210-4689-adce-f822443fecc2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.238848 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.238829    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htfgq\" (UniqueName: \"kubernetes.io/projected/b5607a6f-b210-4689-adce-f822443fecc2-kube-api-access-htfgq\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm\" (UID: \"b5607a6f-b210-4689-adce-f822443fecc2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.326198 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.326113    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:49.378347 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.378209    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"]
Jul 06 22:03:49.383137 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.383108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.389838 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.389655    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-fdf5k\""
Jul 06 22:03:49.394233 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.394204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"]
Jul 06 22:03:49.397941 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.397875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:49.476778 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.476749    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"]
Jul 06 22:03:49.477583 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:03:49.477554    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5607a6f_b210_4689_adce_f822443fecc2.slice/crio-ed6e18d8338da37036c5875fcfe66e7fa7b9c12eb7c3cc5327e54627baf6509b WatchSource:0}: Error finding container ed6e18d8338da37036c5875fcfe66e7fa7b9c12eb7c3cc5327e54627baf6509b: Status 404 returned error can't find the container with id ed6e18d8338da37036c5875fcfe66e7fa7b9c12eb7c3cc5327e54627baf6509b
Jul 06 22:03:49.533452 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.533426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bs4\" (UniqueName: \"kubernetes.io/projected/112b1f94-cac9-40bf-aa75-74c27b4e1c45-kube-api-access-58bs4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.533540 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.533510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/112b1f94-cac9-40bf-aa75-74c27b4e1c45-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.533540 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.533533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112b1f94-cac9-40bf-aa75-74c27b4e1c45-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.634779 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.634697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bs4\" (UniqueName: \"kubernetes.io/projected/112b1f94-cac9-40bf-aa75-74c27b4e1c45-kube-api-access-58bs4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.634933 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.634777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/112b1f94-cac9-40bf-aa75-74c27b4e1c45-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.634933 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.634796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112b1f94-cac9-40bf-aa75-74c27b4e1c45-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.635229 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.635199    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/112b1f94-cac9-40bf-aa75-74c27b4e1c45-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.637366 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.637345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112b1f94-cac9-40bf-aa75-74c27b4e1c45-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.644791 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.644765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bs4\" (UniqueName: \"kubernetes.io/projected/112b1f94-cac9-40bf-aa75-74c27b4e1c45-kube-api-access-58bs4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb\" (UID: \"112b1f94-cac9-40bf-aa75-74c27b4e1c45\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.700527 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.700491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:49.865321 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:49.865291    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"]
Jul 06 22:03:49.866903 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:03:49.866868    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112b1f94_cac9_40bf_aa75_74c27b4e1c45.slice/crio-8f49e062e56575ec63a28ceacd6aa8a4838762d4a3e9b130de91386f842a18b6 WatchSource:0}: Error finding container 8f49e062e56575ec63a28ceacd6aa8a4838762d4a3e9b130de91386f842a18b6: Status 404 returned error can't find the container with id 8f49e062e56575ec63a28ceacd6aa8a4838762d4a3e9b130de91386f842a18b6
Jul 06 22:03:50.193532 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:50.193501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb" event={"ID":"112b1f94-cac9-40bf-aa75-74c27b4e1c45","Type":"ContainerStarted","Data":"2356ea177209361c1b5440acf299b9086af563ad94428c20ef8038708633f04d"}
Jul 06 22:03:50.193994 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:50.193566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:03:50.193994 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:50.193584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb" event={"ID":"112b1f94-cac9-40bf-aa75-74c27b4e1c45","Type":"ContainerStarted","Data":"8f49e062e56575ec63a28ceacd6aa8a4838762d4a3e9b130de91386f842a18b6"}
Jul 06 22:03:50.195242 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:50.195217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm" event={"ID":"b5607a6f-b210-4689-adce-f822443fecc2","Type":"ContainerStarted","Data":"6923bc7e6cce08069327ab7df09d02c1d827f93623c3fa2f2ab11f28d27c2136"}
Jul 06 22:03:50.195368 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:50.195260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm" event={"ID":"b5607a6f-b210-4689-adce-f822443fecc2","Type":"ContainerStarted","Data":"ed6e18d8338da37036c5875fcfe66e7fa7b9c12eb7c3cc5327e54627baf6509b"}
Jul 06 22:03:50.195368 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:50.195305    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:50.245264 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:50.245208    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb" podStartSLOduration=1.245187797 podStartE2EDuration="1.245187797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:03:50.220989888 +0000 UTC m=+5327.555330471" watchObservedRunningTime="2026-07-06 22:03:50.245187797 +0000 UTC m=+5327.579528393"
Jul 06 22:03:51.202751 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:51.202709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm" event={"ID":"b5607a6f-b210-4689-adce-f822443fecc2","Type":"ContainerStarted","Data":"4b0be2e7a466f5d57f0d08321dde9cc77a46d6a768f5139c5b55c7e7a0155f39"}
Jul 06 22:03:51.225514 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:51.225469    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm" podStartSLOduration=3.225453544 podStartE2EDuration="3.225453544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:03:51.223666791 +0000 UTC m=+5328.558007386" watchObservedRunningTime="2026-07-06 22:03:51.225453544 +0000 UTC m=+5328.559794144"
Jul 06 22:03:51.326257 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:51.326218    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:55.933662 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:55.933629    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:56.125836 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.125802    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"]
Jul 06 22:03:56.129735 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.129716    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.133920 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.133896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 06 22:03:56.134026 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.133902    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-kg7g6\""
Jul 06 22:03:56.140800 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.140779    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"]
Jul 06 22:03:56.157413 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.157387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:03:56.196830 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.196734    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.196830 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.196819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.197046 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.196863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d599g\" (UniqueName: \"kubernetes.io/projected/2a53d39a-4a07-49f5-ada7-26c48e5a4541-kube-api-access-d599g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.297927 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.297891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d599g\" (UniqueName: \"kubernetes.io/projected/2a53d39a-4a07-49f5-ada7-26c48e5a4541-kube-api-access-d599g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.298125 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.297949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.298125 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.298076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.298498 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.298467    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.300445 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.300424    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.307228 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.307205    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d599g\" (UniqueName: \"kubernetes.io/projected/2a53d39a-4a07-49f5-ada7-26c48e5a4541-kube-api-access-d599g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.440913 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.440874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:56.585518 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:56.585480    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"]
Jul 06 22:03:56.586265 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:03:56.586240    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a53d39a_4a07_49f5_ada7_26c48e5a4541.slice/crio-aef49de5952720644f3bbc36c2ef361d940f453b2eb987937225119aea240b71 WatchSource:0}: Error finding container aef49de5952720644f3bbc36c2ef361d940f453b2eb987937225119aea240b71: Status 404 returned error can't find the container with id aef49de5952720644f3bbc36c2ef361d940f453b2eb987937225119aea240b71
Jul 06 22:03:57.240450 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:57.240419    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" event={"ID":"2a53d39a-4a07-49f5-ada7-26c48e5a4541","Type":"ContainerStarted","Data":"b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd"}
Jul 06 22:03:57.240866 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:57.240467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" event={"ID":"2a53d39a-4a07-49f5-ada7-26c48e5a4541","Type":"ContainerStarted","Data":"aef49de5952720644f3bbc36c2ef361d940f453b2eb987937225119aea240b71"}
Jul 06 22:03:57.240866 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:57.240524    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:03:57.263570 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:57.263495    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" podStartSLOduration=1.263476598 podStartE2EDuration="1.263476598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:03:57.262691051 +0000 UTC m=+5334.597031647" watchObservedRunningTime="2026-07-06 22:03:57.263476598 +0000 UTC m=+5334.597817192"
Jul 06 22:03:59.326681 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:59.326644    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:59.340752 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:59.340727    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:03:59.345051 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:03:59.345028    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:04:00.269560 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:00.269533    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm"
Jul 06 22:04:10.799399 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:10.799372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:16.275640 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:16.275596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:21.204498 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:21.204468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb"
Jul 06 22:04:27.445019 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:27.444987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:28.247185 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:28.247156    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:04:29.815591 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:29.815553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:29.849702 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:29.849666    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"]
Jul 06 22:04:29.859387 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:04:29.859347    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:04:29.859387 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:04:29.859387 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:04:29.859387 ip-10-0-131-57 kubenswrapper[2582]:  > containerID="b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd" cmd=["/bin/sleep","15"]
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:29.859410    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" message="" err=<
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]:  >
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:04:29.859429    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" podUID="2a53d39a-4a07-49f5-ada7-26c48e5a4541" containerName="main" containerID="cri-o://b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd"
Jul 06 22:04:29.859728 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:29.859487    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" podUID="2a53d39a-4a07-49f5-ada7-26c48e5a4541" containerName="main" containerID="cri-o://b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd" gracePeriod=60
Jul 06 22:04:29.934873 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:29.934840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:04:30.116427 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.116403    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:04:30.229176 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.229146    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d599g\" (UniqueName: \"kubernetes.io/projected/2a53d39a-4a07-49f5-ada7-26c48e5a4541-kube-api-access-d599g\") pod \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") "
Jul 06 22:04:30.229391 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.229252    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tokenizer-uds\") pod \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") "
Jul 06 22:04:30.229391 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.229283    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tls-certs\") pod \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\" (UID: \"2a53d39a-4a07-49f5-ada7-26c48e5a4541\") "
Jul 06 22:04:30.229569 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.229539    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2a53d39a-4a07-49f5-ada7-26c48e5a4541" (UID: "2a53d39a-4a07-49f5-ada7-26c48e5a4541"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:04:30.231457 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.231427    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a53d39a-4a07-49f5-ada7-26c48e5a4541-kube-api-access-d599g" (OuterVolumeSpecName: "kube-api-access-d599g") pod "2a53d39a-4a07-49f5-ada7-26c48e5a4541" (UID: "2a53d39a-4a07-49f5-ada7-26c48e5a4541"). InnerVolumeSpecName "kube-api-access-d599g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:04:30.231457 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.231445    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a53d39a-4a07-49f5-ada7-26c48e5a4541" (UID: "2a53d39a-4a07-49f5-ada7-26c48e5a4541"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:04:30.330938 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.330902    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d599g\" (UniqueName: \"kubernetes.io/projected/2a53d39a-4a07-49f5-ada7-26c48e5a4541-kube-api-access-d599g\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:04:30.330938 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.330935    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tokenizer-uds\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:04:30.330938 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.330944    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a53d39a-4a07-49f5-ada7-26c48e5a4541-tls-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\""
Jul 06 22:04:30.432979 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.432856    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a53d39a-4a07-49f5-ada7-26c48e5a4541" containerID="b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd" exitCode=0
Jul 06 22:04:30.432979 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.432915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"
Jul 06 22:04:30.432979 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.432910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" event={"ID":"2a53d39a-4a07-49f5-ada7-26c48e5a4541","Type":"ContainerDied","Data":"b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd"}
Jul 06 22:04:30.433288 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.433031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm" event={"ID":"2a53d39a-4a07-49f5-ada7-26c48e5a4541","Type":"ContainerDied","Data":"aef49de5952720644f3bbc36c2ef361d940f453b2eb987937225119aea240b71"}
Jul 06 22:04:30.433288 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.433061    2582 scope.go:117] "RemoveContainer" containerID="b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd"
Jul 06 22:04:30.443431 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.443412    2582 scope.go:117] "RemoveContainer" containerID="b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd"
Jul 06 22:04:30.443743 ip-10-0-131-57 kubenswrapper[2582]: E0706 22:04:30.443725    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd\": container with ID starting with b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd not found: ID does not exist" containerID="b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd"
Jul 06 22:04:30.443792 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.443753    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd"} err="failed to get container status \"b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd\": rpc error: code = NotFound desc = could not find container \"b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd\": container with ID starting with b65f622d2aa80ad762c84014b9f8d73ed781f62034cdad19dc9565589719f1bd not found: ID does not exist"
Jul 06 22:04:30.459924 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.459893    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"]
Jul 06 22:04:30.464877 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:30.464841    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm"]
Jul 06 22:04:31.338146 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:04:31.338111    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a53d39a-4a07-49f5-ada7-26c48e5a4541" path="/var/lib/kubelet/pods/2a53d39a-4a07-49f5-ada7-26c48e5a4541/volumes"
Jul 06 22:05:03.853225 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:05:03.853191    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:05:03.874049 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:05:03.874023    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:05:41.428588 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:05:41.428524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:09:09.223687 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:09:09.223653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:09:13.312269 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:09:13.312236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:10:03.894648 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:10:03.894577    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:10:03.912289 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:10:03.912267    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:11:44.924904 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:11:44.924873    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:15:03.932018 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:15:03.931989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:15:03.959413 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:15:03.959387    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:17:55.007060 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:17:55.007029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:20:03.968731 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:20:03.968625    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:20:03.998023 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:20:03.997997    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:21:19.348805 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:21:19.348768    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:22:46.432392 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:22:46.432363    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:22:46.441652 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:22:46.441628    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:22:46.479158 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:22:46.479124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:22:49.138612 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:22:49.138583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:22:49.230942 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:22:49.230908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:22:55.384895 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:22:55.384813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:22:55.484431 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:22:55.484397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:23:05.433234 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:05.433199    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:05.434163 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:05.434147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:05.476039 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:05.476011    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:05.504813 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:05.504787    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:05.564895 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:05.564864    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:07.017307 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:07.017281    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:07.018772 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:07.018751    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:07.047863 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:07.047833    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:07.060257 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:07.060226    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:07.121115 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:07.121083    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:08.554503 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:08.554475    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:08.555785 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:08.555768    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:08.583601 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:08.583577    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:08.595971 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:08.595941    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:08.653600 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:08.653568    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:10.064196 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:10.064165    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:10.065544 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:10.065525    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:10.096550 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:10.096523    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:10.110183 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:10.110161    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:10.169396 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:10.169367    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:11.665184 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:11.665150    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:11.666784 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:11.666766    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:11.699120 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:11.699091    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:11.711800 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:11.711781    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:11.774410 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:11.774369    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:13.304092 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:13.304045    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:13.305826 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:13.305805    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:13.342217 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:13.342190    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:13.356109 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:13.356081    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:13.419016 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:13.418986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:14.884881 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:14.884849    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:14.886314 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:14.886296    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:14.925228 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:14.925202    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:14.945275 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:14.945249    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:15.018028 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:15.018000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:16.511556 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:16.511527    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:16.512949 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:16.512920    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:16.544061 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:16.544032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:16.558834 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:16.558809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:16.625639 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:16.625611    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:18.055105 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:18.055070    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:18.056682 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:18.056657    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:18.089723 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:18.089693    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:18.103860 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:18.103835    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:18.165267 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:18.165240    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:19.566449 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:19.566421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:19.567928 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:19.567910    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:19.597428 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:19.597403    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:19.610123 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:19.610101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:19.669612 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:19.669580    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:21.116253 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:21.116223    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:21.117578 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:21.117561    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:21.150953 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:21.150922    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:21.163877 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:21.163846    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:21.221621 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:21.221590    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:22.635833 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:22.635797    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:22.637288 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:22.637260    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:22.667015 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:22.666989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:22.681402 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:22.681368    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:22.738406 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:22.738361    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:24.137850 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:24.137817    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:24.139381 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:24.139360    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:24.170321 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:24.170294    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:24.191642 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:24.191610    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:24.252727 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:24.252697    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:25.677181 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:25.677148    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:25.678574 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:25.678558    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:25.708916 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:25.708890    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:25.724521 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:25.724493    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:25.782636 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:25.782606    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:27.238030 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:27.238002    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:27.239619 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:27.239597    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:27.271465 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:27.271439    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:27.287943 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:27.287922    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:27.358611 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:27.358582    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:28.840807 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:28.840769    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/2.log"
Jul 06 22:23:28.842500 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:28.842475    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-799f46c59b-pl8dd_ee146378-761d-47dd-96b5-24f9cfaff899/istio-proxy/3.log"
Jul 06 22:23:28.879741 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:28.879720    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/main/0.log"
Jul 06 22:23:28.900181 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:28.900147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm_b5607a6f-b210-4689-adce-f822443fecc2/llm-d-routing-sidecar/0.log"
Jul 06 22:23:28.965526 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:28.965500    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb_112b1f94-cac9-40bf-aa75-74c27b4e1c45/main/0.log"
Jul 06 22:23:31.531440 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:31.531404    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/13.log"
Jul 06 22:23:31.532818 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:31.532795    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/14.log"
Jul 06 22:23:31.545103 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:31.545077    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757d9b65f5-7xf7w_30cb12a1-e55d-432b-b856-258d10383502/router/0.log"
Jul 06 22:23:33.628308 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:33.628281    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/13.log"
Jul 06 22:23:33.630233 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:33.630215    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/14.log"
Jul 06 22:23:33.644600 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:33.644576    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757d9b65f5-7xf7w_30cb12a1-e55d-432b-b856-258d10383502/router/0.log"
Jul 06 22:23:35.731037 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:35.731004    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/13.log"
Jul 06 22:23:35.732891 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:35.732865    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/14.log"
Jul 06 22:23:35.747618 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:35.747600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757d9b65f5-7xf7w_30cb12a1-e55d-432b-b856-258d10383502/router/0.log"
Jul 06 22:23:37.828404 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:37.828355    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/13.log"
Jul 06 22:23:37.830264 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:37.830234    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/14.log"
Jul 06 22:23:37.843069 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:37.843044    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757d9b65f5-7xf7w_30cb12a1-e55d-432b-b856-258d10383502/router/0.log"
Jul 06 22:23:38.810316 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:38.810289    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:38.823234 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:38.823209    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:38.840913 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:38.840888    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:38.855349 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:38.855322    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:39.080120 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:39.080033    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:39.111425 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:39.111397    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:40.115744 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:40.115710    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:40.131713 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:40.131686    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:40.145766 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:40.145739    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:40.158767 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:40.158746    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:40.383379 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:40.383299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:40.423218 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:40.423190    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:41.467986 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:41.467932    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:41.483086 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:41.483061    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:41.497981 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:41.497941    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:41.511476 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:41.511451    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:41.745207 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:41.745112    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:41.787524 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:41.787482    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:42.796628 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:42.796598    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:42.809533 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:42.809506    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:42.823528 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:42.823501    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:42.836837 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:42.836814    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:43.060647 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:43.060527    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:43.090830 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:43.090800    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:44.118715 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:44.118680    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:44.132116 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:44.132092    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:44.149825 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:44.149789    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:44.166000 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:44.165975    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:44.385920 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:44.385827    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:44.451230 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:44.451200    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:45.464196 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:45.464156    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:45.477445 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:45.477409    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:45.494911 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:45.494889    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:45.509513 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:45.509489    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:45.731176 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:45.731068    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:45.767532 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:45.767501    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:46.770480 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:46.770448    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:46.786749 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:46.786710    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:46.800604 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:46.800574    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:46.813319 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:46.813289    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:47.034606 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:47.034519    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:47.070071 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:47.070027    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:48.153026 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:48.152986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:48.168021 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:48.167996    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:48.182365 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:48.182331    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:48.196201 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:48.196179    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:48.421583 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:48.421504    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:48.452006 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:48.451956    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:49.440905 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:49.440855    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:49.458899 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:49.458870    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:49.472074 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:49.472056    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:49.485508 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:49.485485    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:49.712301 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:49.712219    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:49.744100 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:49.744073    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:50.765020 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:50.764985    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:50.779858 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:50.779834    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:50.794885 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:50.794862    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:50.808220 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:50.808199    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:51.044085 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:51.044003    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:51.073611 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:51.073580    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:52.083957 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:52.083929    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:23:52.100115 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:52.100089    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:23:52.113735 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:52.113711    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:23:52.125585 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:52.125564    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:23:52.348365 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:52.348260    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:23:52.380866 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:52.380827    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:23:54.940203 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:54.940170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:23:56.570122 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:56.570084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:23:58.237860 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:58.237823    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w86tz_042b84ca-928d-462f-8a34-fb128944ed65/global-pull-secret-syncer/0.log"
Jul 06 22:23:58.282324 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:58.282301    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log"
Jul 06 22:23:58.449241 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:23:58.449206    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-57.ec2.internal_f22a2c57117da9e2036a061b6a5bcbad/haproxy/0.log"
Jul 06 22:24:01.383914 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:01.383879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:24:01.446590 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:01.446555    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:24:03.148203 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:03.148111    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z4dhg_452f3ebb-1fc9-4ea2-80c1-bd0508dc7708/authorino/0.log"
Jul 06 22:24:03.175329 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:03.175303    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-gmblh_5caebcdc-62e3-46d8-a927-d0b968960429/manager/0.log"
Jul 06 22:24:03.214397 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:03.214358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-wxznh_5ef4aac2-3fc7-46b5-8942-a8a7e46d8841/manager/0.log"
Jul 06 22:24:03.239928 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:03.239903    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-7jl5x_2dd7826b-f568-4d72-8b11-c660ce6ea932/kuadrant-console-plugin/0.log"
Jul 06 22:24:03.483319 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:03.483273    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw_eff329fb-f3db-4c74-816f-a5882742d6f0/manager/0.log"
Jul 06 22:24:03.538942 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:03.538899    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-s2frj_e8884f9b-a492-4efa-91c2-a79975e1b0ea/manager/0.log"
Jul 06 22:24:04.853303 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:04.853266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de04391c-d7e4-4837-b02d-e228888e79e5/alertmanager/0.log"
Jul 06 22:24:04.878247 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:04.878225    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de04391c-d7e4-4837-b02d-e228888e79e5/config-reloader/0.log"
Jul 06 22:24:04.902355 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:04.902335    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de04391c-d7e4-4837-b02d-e228888e79e5/kube-rbac-proxy-web/0.log"
Jul 06 22:24:04.925164 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:04.925143    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de04391c-d7e4-4837-b02d-e228888e79e5/kube-rbac-proxy/0.log"
Jul 06 22:24:04.948463 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:04.948437    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de04391c-d7e4-4837-b02d-e228888e79e5/kube-rbac-proxy-metric/0.log"
Jul 06 22:24:04.971115 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:04.971094    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de04391c-d7e4-4837-b02d-e228888e79e5/prom-label-proxy/0.log"
Jul 06 22:24:04.996832 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:04.996809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de04391c-d7e4-4837-b02d-e228888e79e5/init-config-reloader/0.log"
Jul 06 22:24:05.172443 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:05.172359    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86bddcdcdf-lprv2_4162691b-9b53-4f69-b26e-c10d6e157587/metrics-server/0.log"
Jul 06 22:24:05.198996 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:05.198959    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-wzr47_a033849b-0d61-42a5-9aa8-34f6b749b22f/monitoring-plugin/0.log"
Jul 06 22:24:05.235228 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:05.235204    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vj4l_e02d2f7b-67e1-4405-8c76-a22f6247ab73/node-exporter/0.log"
Jul 06 22:24:05.259611 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:05.259585    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vj4l_e02d2f7b-67e1-4405-8c76-a22f6247ab73/kube-rbac-proxy/0.log"
Jul 06 22:24:05.289030 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:05.289008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vj4l_e02d2f7b-67e1-4405-8c76-a22f6247ab73/init-textfile/0.log"
Jul 06 22:24:06.002262 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.002225    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-746978fcf7-tlq6x_1eb84500-930c-4c37-8942-6a27e39ff50c/telemeter-client/0.log"
Jul 06 22:24:06.027889 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.027866    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-746978fcf7-tlq6x_1eb84500-930c-4c37-8942-6a27e39ff50c/reload/0.log"
Jul 06 22:24:06.050767 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.050742    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-746978fcf7-tlq6x_1eb84500-930c-4c37-8942-6a27e39ff50c/kube-rbac-proxy/0.log"
Jul 06 22:24:06.079869 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.079848    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687687fc7d-drjss_fa0ab60a-9947-406c-9495-7f4cf0512c8b/thanos-query/0.log"
Jul 06 22:24:06.102003 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.101975    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687687fc7d-drjss_fa0ab60a-9947-406c-9495-7f4cf0512c8b/kube-rbac-proxy-web/0.log"
Jul 06 22:24:06.131000 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.130956    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687687fc7d-drjss_fa0ab60a-9947-406c-9495-7f4cf0512c8b/kube-rbac-proxy/0.log"
Jul 06 22:24:06.156728 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.156701    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687687fc7d-drjss_fa0ab60a-9947-406c-9495-7f4cf0512c8b/prom-label-proxy/0.log"
Jul 06 22:24:06.180647 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.180613    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687687fc7d-drjss_fa0ab60a-9947-406c-9495-7f4cf0512c8b/kube-rbac-proxy-rules/0.log"
Jul 06 22:24:06.203458 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:06.203414    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-687687fc7d-drjss_fa0ab60a-9947-406c-9495-7f4cf0512c8b/kube-rbac-proxy-metrics/0.log"
Jul 06 22:24:08.031938 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.031908    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/1.log"
Jul 06 22:24:08.036700 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.036682    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-pn8bl_fd23fc80-e745-4494-99c4-4dde8a5772f0/console-operator/2.log"
Jul 06 22:24:08.344157 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.344076    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"]
Jul 06 22:24:08.344504 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.344492    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a53d39a-4a07-49f5-ada7-26c48e5a4541" containerName="main"
Jul 06 22:24:08.344551 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.344505    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a53d39a-4a07-49f5-ada7-26c48e5a4541" containerName="main"
Jul 06 22:24:08.344586 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.344576    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a53d39a-4a07-49f5-ada7-26c48e5a4541" containerName="main"
Jul 06 22:24:08.347632 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.347607    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.350198 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.350160    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hm2m\"/\"kube-root-ca.crt\""
Jul 06 22:24:08.350331 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.350208    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hm2m\"/\"openshift-service-ca.crt\""
Jul 06 22:24:08.350331 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.350280    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hm2m\"/\"default-dockercfg-k7bqq\""
Jul 06 22:24:08.356262 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.356243    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"]
Jul 06 22:24:08.456550 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.456518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-proc\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.456718 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.456579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-lib-modules\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.456718 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.456606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-podres\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.456718 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.456629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-sys\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.456827 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.456723    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnk6p\" (UniqueName: \"kubernetes.io/projected/1cf95a61-a552-4f6a-92bc-00f522386c6b-kube-api-access-hnk6p\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557246 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-sys\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557441 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnk6p\" (UniqueName: \"kubernetes.io/projected/1cf95a61-a552-4f6a-92bc-00f522386c6b-kube-api-access-hnk6p\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557441 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-proc\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557441 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-lib-modules\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557441 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557355    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-sys\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557441 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-podres\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557441 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557431    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-proc\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557665 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557503    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-podres\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.557665 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.557503    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cf95a61-a552-4f6a-92bc-00f522386c6b-lib-modules\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.566283 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.566253    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnk6p\" (UniqueName: \"kubernetes.io/projected/1cf95a61-a552-4f6a-92bc-00f522386c6b-kube-api-access-hnk6p\") pod \"perf-node-gather-daemonset-vx28g\" (UID: \"1cf95a61-a552-4f6a-92bc-00f522386c6b\") " pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.658177 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.658089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.792336 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.792308    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"]
Jul 06 22:24:08.793929 ip-10-0-131-57 kubenswrapper[2582]: W0706 22:24:08.793899    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1cf95a61_a552_4f6a_92bc_00f522386c6b.slice/crio-62fd5c98a5e5feb0ea9f419420344c6a3f1b4a3c6d621acc69f60de2c3da2187 WatchSource:0}: Error finding container 62fd5c98a5e5feb0ea9f419420344c6a3f1b4a3c6d621acc69f60de2c3da2187: Status 404 returned error can't find the container with id 62fd5c98a5e5feb0ea9f419420344c6a3f1b4a3c6d621acc69f60de2c3da2187
Jul 06 22:24:08.938201 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.938171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g" event={"ID":"1cf95a61-a552-4f6a-92bc-00f522386c6b","Type":"ContainerStarted","Data":"91372a18c6af6f76e887f89264e8f371f38e503b77059a77bf7d69596b630e61"}
Jul 06 22:24:08.938343 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.938219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g" event={"ID":"1cf95a61-a552-4f6a-92bc-00f522386c6b","Type":"ContainerStarted","Data":"62fd5c98a5e5feb0ea9f419420344c6a3f1b4a3c6d621acc69f60de2c3da2187"}
Jul 06 22:24:08.938343 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.938296    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:08.958515 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:08.958466    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g" podStartSLOduration=0.958452092 podStartE2EDuration="958.452092ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:08.955456979 +0000 UTC m=+6546.289797594" watchObservedRunningTime="2026-07-06 22:24:08.958452092 +0000 UTC m=+6546.292792686"
Jul 06 22:24:09.168466 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:09.168429    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-sklbl_02624e11-4c9e-4ec1-be35-7ce3a74c65af/volume-data-source-validator/0.log"
Jul 06 22:24:10.289019 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:10.288987    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rps8j_05d6c585-6fc0-45f4-a258-e59b0d234bf9/dns/0.log"
Jul 06 22:24:10.312229 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:10.312200    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rps8j_05d6c585-6fc0-45f4-a258-e59b0d234bf9/kube-rbac-proxy/0.log"
Jul 06 22:24:10.404550 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:10.404520    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hx4xq_09d366b1-0bfa-49f1-8cfe-d68f9fd5dece/dns-node-resolver/0.log"
Jul 06 22:24:11.019277 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:11.019245    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55cdb97ff5-g8zn2_73657f86-99d3-4f35-8bd4-a9fc7b56c977/registry/0.log"
Jul 06 22:24:11.047291 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:11.047266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4jcp4_700e472f-98d4-4970-b9ea-3b1d6ed575e6/node-ca/0.log"
Jul 06 22:24:13.225474 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:13.225446    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/13.log"
Jul 06 22:24:13.227410 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:13.227392    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/14.log"
Jul 06 22:24:13.248763 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:13.248743    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757d9b65f5-7xf7w_30cb12a1-e55d-432b-b856-258d10383502/router/0.log"
Jul 06 22:24:13.818141 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:13.818115    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vr2td_a3c1ec3f-6328-48cc-bc69-d20eaab9e41c/serve-healthcheck-canary/0.log"
Jul 06 22:24:14.491030 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:14.491001    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fl8wb_d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8/kube-rbac-proxy/0.log"
Jul 06 22:24:14.515536 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:14.515508    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fl8wb_d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8/exporter/0.log"
Jul 06 22:24:14.538458 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:14.538432    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fl8wb_d02ba9b3-20a9-4423-82ae-2c6fb40c4cd8/extractor/0.log"
Jul 06 22:24:14.952714 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:14.952687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8hm2m/perf-node-gather-daemonset-vx28g"
Jul 06 22:24:18.389569 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:18.389535    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-68bc99fcb8-7bghc_6b250379-0cfd-4a78-98db-a83fdc041069/manager/0.log"
Jul 06 22:24:18.456469 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:18.456439    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-s8blt_792e2b7d-169d-4b8f-9365-2126f95ab769/server/0.log"
Jul 06 22:24:18.971906 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:18.971875    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-zlrl9_9915cda1-77b9-45e8-9882-feba99d6a970/manager/0.log"
Jul 06 22:24:19.007112 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:19.007086    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vmx7n_6d05dfe9-43fd-4f2c-b62e-125fd10adce5/s3-init/0.log"
Jul 06 22:24:19.026272 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:19.026206    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vmx7n_6d05dfe9-43fd-4f2c-b62e-125fd10adce5/download-hf-model/0.log"
Jul 06 22:24:20.582550 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:20.582516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hpfr_fef030ed-1788-4ced-acbc-6b11fb567dd5/konnectivity-agent/0.log" limit=1024
Jul 06 22:24:21.117606 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:21.117578    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/13.log"
Jul 06 22:24:21.119171 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:21.119150    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/14.log"
Jul 06 22:24:21.133108 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:21.133079    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757d9b65f5-7xf7w_30cb12a1-e55d-432b-b856-258d10383502/router/0.log"
Jul 06 22:24:23.227194 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:23.227162    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/13.log"
Jul 06 22:24:23.228853 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:23.228833    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-vmnww_dfe3aa62-0935-48f7-a93d-8d21917d35a5/istio-proxy/14.log"
Jul 06 22:24:23.244824 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:23.244800    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757d9b65f5-7xf7w_30cb12a1-e55d-432b-b856-258d10383502/router/0.log"
Jul 06 22:24:24.653761 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:24.653686    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-97pd6_61dd7738-53ef-4ece-a8dd-ca81fa6557df/migrator/0.log"
Jul 06 22:24:24.677795 ip-10-0-131-57 kubenswrapper[2582]: I0706 22:24:24.677769    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-97pd6_61dd7738-53ef-4ece-a8dd-ca81fa6557df/graceful-termination/0.log"